Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28822

System Test : Rebalance fails when rebalancing out 2 indexer nodes together

    XMLWordPrintable

Details

    Description

      Build : 5.5.0-2256

      In the component level system test for secondary indexing, at a step where we rebalance out 2 indexer nodes together, rebalance fails with the following error. KV ops and queries were concurrently going on at that time. I did not see any panics in the indexer logs on any indexer node.

      [ns_server:error,2018-03-21T02:15:09.952-07:00,ns_1@172.23.104.16:service_rebalancer-index<0.12360.8>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
      		 
      [{'ns_1@172.23.104.23',nack}]
      		 
      [user:error,2018-03-21T02:15:09.953-07:00,ns_1@172.23.104.16:<0.9450.0>:ns_orchestrator:do_log_rebalance_completion:1122]Rebalance exited with reason {service_rebalance_failed,index,
      		 
                                    {rebalance_failed,
      		 
                                     {service_error,
      		 
                                      <<"Post http://127.0.0.1:9102/dropIndex: net/http: request canceled (Client.Timeout exceeded while awaiting headers)">>}}}

      Rebalance out for single node worked fine.

      172.23.104.21,172.23.104.94 are the indexer nodes being removed. 172.23.104.16 is the master node.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-30577 [Backport MB-28822] - System Test : Rebalance fails when rebalancing out 2 indexer nodes together

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          For Gerrit Dashboard: MB-28822
          # Subject Branch Project Status CR V
          91378,1 MB-28822: Delete index in background during rebalancing unstable indexing Status: ABANDONED 0 0
          91445,4 MB-28822: Do not drop index until index build is done on local node during rebalance unstable indexing Status: MERGED +2 +1

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty