Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51046

[system test] FTS: high fragmentation causing disk full in one of the kv node

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      Build: 7.1.0 build 2304
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
      Scale: 1
      Test Cycle: 6
      Cluster - comp cluster 2

      KV bucket fragmentation seen high in one of the node (172.23.106.154) and seen disk full on same node and below errors:

      2022-02-16T22:41:32.721-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket4" on node 172.23.106.154.
      		 
      2022-02-16T22:41:32.722-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket3" on node 172.23.106.154.
      		 
      2022-02-16T22:42:32.722-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket4" on node 172.23.106.154.
      		 
      2022-02-16T22:42:32.723-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket3" on node 172.23.106.154.
      		 
      2022-02-16T22:43:32.723-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket4" on node 172.23.106.154.
      		 
      2022-02-16T22:43:32.724-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket3" on node 172.23.106.154.
      		 
      2022-02-16T22:44:32.724-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket4" on node 172.23.106.154.
      		 
      2022-02-16T22:44:32.725-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket3" on node 172.23.106.154.
      		 
      2022-02-16T22:45:32.727-08:00, menelaus_web_alerts_srv:0:info:message(ns_1@172.23.106.154) - Write Commit Failure. Disk write failed for item in Bucket "bucket4" on node 172.23.106.154.


      Test step around the time when the errors started:

      [2022-02-16T22:35:12-08:00, sequoiatools/couchbase-cli:7.1:bb883d] server-add -c 172.23.106.253:8091 --server-add https://172.23.105.186 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts
      		 
      [pull] sequoiatools/couchbase-cli:7.1
      		 
      [2022-02-16T22:35:24-08:00, sequoiatools/couchbase-cli:7.1:8a13cd] rebalance -c 172.23.106.253:8091 -u Administrator -p password
      		 
      [pull] sequoiatools/cmd
      		 
      [2022-02-16T22:38:47-08:00, sequoiatools/cmd:4e1c54] 60
      		 
      [pull] sequoiatools/cmd
      		 
      [2022-02-16T22:39:55-08:00, sequoiatools/cmd:99f181] 900

      Logs around the time the above errors started:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.105.185.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.105.186.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.105.190.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.106.154.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.106.182.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.106.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.106.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.106.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.106.255.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.107.89.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.97.213.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1645083830/collectinfo-2022-02-17T074352-ns_1%40172.23.97.214.zip

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-50941 Compaction reschedules but never runs (sleeping for ~0 seconds?)

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty