Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-28822

System Test : Rebalance fails when rebalancing out 2 indexer nodes together

    XMLWordPrintable

Details

    Description

      Build : 5.5.0-2256

      In the component level system test for secondary indexing, at a step where we rebalance out 2 indexer nodes together, rebalance fails with the following error. KV ops and queries were concurrently going on at that time. I did not see any panics in the indexer logs on any indexer node.

      [ns_server:error,2018-03-21T02:15:09.952-07:00,ns_1@172.23.104.16:service_rebalancer-index<0.12360.8>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
      [{'ns_1@172.23.104.23',nack}]
      [user:error,2018-03-21T02:15:09.953-07:00,ns_1@172.23.104.16:<0.9450.0>:ns_orchestrator:do_log_rebalance_completion:1122]Rebalance exited with reason {service_rebalance_failed,index,
                                    {rebalance_failed,
                                     {service_error,
                                      <<"Post http://127.0.0.1:9102/dropIndex: net/http: request canceled (Client.Timeout exceeded while awaiting headers)">>}}}
      

      Rebalance out for single node worked fine.

      172.23.104.21,172.23.104.94 are the indexer nodes being removed. 172.23.104.16 is the master node.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty