Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33451

[System Test] plasma rollback can take a long time

    XMLWordPrintable

Details

    Description

      Build : 6.5.0-2647
      Test : -test tests/integration/test_allFeatures_alice_timers.yml -scope tests/integration/scope_Xattrs_Alice.yml
      Scale : 3
      Iteration : 1

      There is a step in the test to rebalance out a failed over kv node and add a new kv node to the cluster. This rebalance fails because of index service.

      [ns_server:error,2019-03-20T14:55:44.960-07:00,ns_1@172.23.108.103:service_rebalancer-index<0.18446.474>:service_rebalancer:run_rebalance:82]Agent terminated during the rebalance: {'DOWN',
      		 
                                              #Ref<0.2244132595.2484600835.105926>,
      		 
                                              process,<22559.18956.224>,
      		 
                                              {linked_process_died,
      		 
                                               <22559.28594.226>,
      		 
                                               {timeout,
      		 
                                                {gen_server,call,
      		 
                                                 [<22559.22183.224>,
      		 
                                                  {call,"ServiceAPI.GetTaskList",
      		 
                                                   #Fun<json_rpc_connection.0.102434519>},
      		 
                                                  60000]}}}}
      		 
      [ns_server:error,2019-03-20T14:55:45.003-07:00,ns_1@172.23.108.103:service_rebalancer-index<0.18446.474>:service_agent:process_bad_results:810]Service call unset_rebalancer (service index) failed on some nodes:
      		 
      [{'ns_1@172.23.96.56',nack}]
      		 
      [user:error,2019-03-20T14:55:45.005-07:00,ns_1@172.23.108.103:<0.12272.0>:ns_orchestrator:do_log_rebalance_completion:1206]Rebalance exited with reason {service_rebalance_failed,index,
      		 
                                    {linked_process_died,<22559.28594.226>,
      		 
                                     {timeout,
      		 
                                      {gen_server,call,
      		 
                                       [<22559.22183.224>,
      		 
                                        {call,"ServiceAPI.GetTaskList",
      		 
                                         #Fun<json_rpc_connection.0.102434519>},
      		 
                                        60000]}}}}. Operation Id = 55b509718bc155930e0f282292e19569

      Cluster config :
      index : 5 ===== > [172.23.104.164:8091 172.23.104.67:8091 172.23.104.88:8091 172.23.96.252:8091 172.23.96.56:8091]
      n1ql : 2 ===== > [172.23.104.69:8091 172.23.104.70:8091]
      cbas : 2 ===== > [172.23.106.188:8091 172.23.96.168:8091]
      kv : 9 ===== > [172.23.108.103:8091 172.23.108.104:8091 172.23.96.145:8091 172.23.96.148:8091 172.23.97.238:8091 172.23.97.239:8091 172.23.99.20:8091 172.23.99.21:8091 172.23.99.25:8091]
      fts : 2 ===== > [172.23.96.253:8091 172.23.96.95:8091]
      eventing : 2 ===== > [172.23.97.242:8091 172.23.98.135:8091]

      Attachments

        For Gerrit Dashboard: MB-33451
        # Subject Branch Project Status CR V
        106662,1 MB-33451 Change log level for stream reader message unstable indexing Status: MERGED +2 +1
        113911,3 MB-33451 allow create/build index during recovery unstable indexing Status: MERGED +2 +1

        Activity

          People

            deepkaran.salooja Deepkaran Salooja
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty