Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50713

[FTS] Rebalances failing with service_rebalance_failed,fts - inactivity_timeout

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 7.1.0
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-2197

      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
      Scale: 1

      Rebalances failing with below:

      2022-01-31T17:30:08.162-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.106.253) - Rebalance exited with reason {service_rebalance_failed,fts,
      		 
                                    {worker_died,
      		 
                                     {'EXIT',<0.12299.117>,
      		 
                                      {rebalance_failed,inactivity_timeout}}}}.
      		 
      Rebalance Operation Id = 8129c46d432460c8b46dfa9957ed40a3

      Test steps where 2 rebalances failed:

      1: Failover / rebalance of 172.23.106.154

      [2022-01-31T16:47:51-08:00, sequoiatools/couchbase-cli:7.1:5c1c61] failover -c 172.23.106.253:8091 --server-failover 172.23.106.154:8091 -u Administrator -p password --hard
      		 
      [pull] sequoiatools/couchbase-cli:7.1
      		 
      [2022-01-31T16:48:00-08:00, sequoiatools/couchbase-cli:7.1:0735f0] rebalance -c 172.23.106.253:8091 -u Administrator -p password
      		 
      [2022-01-31T16:47:51-08:00, sequoiatools/couchbase-cli:7.1:5c1c61] failover -c 172.23.106.253:8091 --server-failover 172.23.106.154:8091 -u Administrator -p password --hard
      		 
      [pull] sequoiatools/couchbase-cli:7.1
      		 
      [2022-01-31T16:48:00-08:00, sequoiatools/couchbase-cli:7.1:0735f0] rebalance -c 172.23.106.253:8091 -u Administrator -p password

      2. Failover/recovery/rebalance of 172.23.105.186

      [2022-01-31T17:46:35-08:00, sequoiatools/couchbase-cli:7.1:a57fc0] failover -c 172.23.106.253:8091 --server-failover 172.23.105.186:8091 -u Administrator -p password --hard
      		 
      [pull] sequoiatools/couchbase-cli:7.1
      		 
      [2022-01-31T17:46:44-08:00, sequoiatools/couchbase-cli:7.1:1159b3] recovery -c 172.23.106.253:8091 --server-recovery 172.23.105.186:8091 --recovery-type full -u Administrator -p password
      		 
      [pull] sequoiatools/couchbase-cli:7.1
      		 
      [2022-01-31T17:46:49-08:00, sequoiatools/couchbase-cli:7.1:28880f] rebalance -c 172.23.106.253:8091 -u Administrator -p password
      		 
      →  
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.106.253:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs 28880f
      		 
      docker start 28880f
      		 
       
      		 
      *Unable to display progress bar on this os
      		 
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      		 
      [pull] sequoiatools/cmd
      		 
      [2022-01-31T17:57:10-08:00, sequoiatools/cmd:b2c957] 60

      Failure log:

      172.23.106.253 :
      		 
      [user:error,2022-01-31T16:58:04.497-08:00,ns_1@172.23.106.253:<0.8982.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {service_rebalance_failed,fts,
      		 
      [user:error,2022-01-31T17:30:08.162-08:00,ns_1@172.23.106.253:<0.8982.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {service_rebalance_failed,fts,

      Cluster spec:

      ########## Cluster config ##################
      		 
      ######  n1ql : 2 ===== > [172.23.105.185:8091 172.23.106.182:8091]  ###########
      		 
      ######  fts : 5 ===== > [172.23.105.186:8091 172.23.105.190:8091 172.23.106.154:8091 172.23.106.255:8091 172.23.97.213:8091]  ###########
      		 
      ######  kv : 4 ===== > [172.23.106.242:8091 172.23.106.243:8091 172.23.106.253:8091 172.23.107.89:8091]  ###########

      Logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.105.185.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.105.186.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.105.190.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.106.154.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.106.182.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.106.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.106.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.106.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.106.255.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.107.89.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.97.213.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1643680587/collectinfo-2022-02-01T015628-ns_1%40172.23.97.214.zip

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-51334 [System Test]Rebalance exited with reason {service_rebalance_failed,fts - worker_died - inactivity_timeout

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          For Gerrit Dashboard: MB-50713
          # Subject Branch Project Status CR V
          169847,2 adding seqChecksTimeoutInSec for issue: MB-50713 master sequoia Status: MERGED +2 +1
          169881,3 MB-50713 - Rebalance failures due to inactivity_timeout master cbgt Status: MERGED +2 +1

          Activity

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty