Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60709

[System Test][MOI] Rebalance failure - http://127.0.0.1:9102/dropIndex: context deadline exceeded

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • Morpheus
    • 7.6.0
    • secondary-index
    • 7.6.0-2095
    • Untriaged
    • 0
    • Unknown

    Description

      It appears that the rebalance has failed with the following message -

      [ns_server:error,2024-02-06T12:48:26.585-08:00,ns_1@172.23.218.84:service_manager-index<0.26641.166>:service_manager:run_op_worker:219]Worker terminated abnormally: {'EXIT',<0.26804.166>,
                                     {task_failed,rebalance,
                                      {service_error,
                                       <<"Post \"http://127.0.0.1:9102/dropIndex\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}}
      [ns_server:info,2024-02-06T12:48:26.587-08:00,ns_1@172.23.218.84:rebalance_agent<0.29522.0>:rebalance_agent:handle_down:290]Rebalancer process <0.24860.166> died (reason {service_rebalance_failed,
                                                     index,
                                                     {worker_died,
                                                      {'EXIT',<0.26804.166>,
                                                       {task_failed,rebalance,
                                                        {service_error,
                                                         <<"Post \"http://127.0.0.1:9102/dropIndex\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}}}}).
      [user:error,2024-02-06T12:48:26.591-08:00,ns_1@172.23.218.84:<0.30057.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index,
                                    {worker_died,
                                     {'EXIT',<0.26804.166>,
                                      {task_failed,rebalance,
                                       {service_error,
                                        <<"Post \"http://127.0.0.1:9102/dropIndex\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)">>}}}}}.
      

      Seen during iteration 12. Auto-failover was enabled about 20 minutes prior to the rebalance failure and the test triggered an autofailover by stop couchbase server on node 218.87

      [2024-02-06T12:20:37-08:00, sequoiatools/cmd:6bc9fe] 300
      [2024-02-06T12:26:26-08:00, sequoiatools/couchbase-cli:7.6:29d1dc] setting-autofailover -c 172.23.218.86:8091 -u Administrator -p password --enable-auto-failover=1 --auto-failover-timeout=30 --max-failovers=1
      [2024-02-06T12:26:32-08:00, sequoiatools/cmd:08e601] 10
      [2024-02-06T12:26:48-08:00, sequoiatools/cbutil:d93355] /cbinit.py 172.23.218.87 root couchbase stop
      [2024-02-06T12:26:54-08:00, sequoiatools/cmd:353888] 40
      [2024-02-06T12:27:40-08:00, sequoiatools/couchbase-cli:7.6:53f970] rebalance -c 172.23.218.86:8091 -u Administrator -p password
       
      Error occurred on container - sequoiatools/couchbase-cli:7.6:[rebalance -c 172.23.218.86:8091 -u Administrator
      

      Cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.120.202.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.83.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.84.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.85.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.87.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.218.89.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.96.207.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707253208/collectinfo-2024-02-06T211024-ns_1%40172.23.96.231.zip

      Attachments

        Activity

          People

            dhananjay.kshirsagar Dhananjay Kshirsagar
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:

              PagerDuty