Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37167

Rebalance multiple kv nodes failed with Some apps are deploying or resuming on some or all Eventing nodes

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      build: 6.5.0-4917 , last passed on 6.5.0-4908

      • Create 5 node cluster 3 kv , 1 eventing , 1 index and n1ql
      • Deploy SBM handler with timer
      • Start data loading in source bucket
      • Add 2 more kv nodes – failed
      • Verify results in source bucket
      • Remove above added nodes from the cluster

      ./testrunner -i /tmp/testexec.17696.ini -p get-cbcollect-info=True,GROUP=source_bucket_mutation_timers -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,handler_code=source_bucket_mutation_with_timers,source_bucket_mutation=True,GROUP=source_bucket_mutation_timers 

      2019-12-04 21:39:56,138] - [rest_client:3331] ERROR - {u'node': u'ns_1@172.23.104.238', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n                              {worker_died,\n                               {\'EXIT\',<0.29602.2>,\n                                {{badmatch,\n                                  {error,\n                                   {unknown_error,\n                                    <<"Some apps are deploying or resuming on some or all Eventing nodes">>}}},\n                                 [{service_rebalancer,rebalance_worker,1,\n                                   [{file,"src/service_rebalancer.erl"},\n                                    {line,170}]},\n                                  {proc_lib,init_p,3,\n                                   [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 6868fa7d4f59e481b6b7553fb2b8c450', u'shortText': u'message', u'serverTime': u'2019-12-04T21:39:48.893Z', u'module': u'ns_orchestrator', u'tstamp': 1575524388893, u'type': u'critical'}
       
       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            lynn.straus Lynn Straus added a comment -

            Multiple changes coming in this area with CCBC-1130 which has ETA of Dec 12.  Setting ETA for this dependent ticket to Dec 12.

            lynn.straus Lynn Straus added a comment - Multiple changes coming in this area with CCBC-1130 which has ETA of Dec 12.  Setting ETA for this dependent ticket to Dec 12.

            See comments on MB-37197 about status update. Currently both of these are in the same area and could possibly be the same. Continuing to work to isolate. Updating the "due" date accordingly.

            ingenthr Matt Ingenthron added a comment - See comments on MB-37197 about status update. Currently both of these are in the same area and could possibly be the same. Continuing to work to isolate. Updating the "due" date accordingly.

            Suraj Naik, could you re-run the test in the subject? Or it is the same tests that been running for MB-37197?

            avsej Sergey Avseyev added a comment - Suraj Naik , could you re-run the test in the subject? Or it is the same tests that been running for MB-37197 ?

            Sergey Avseyev,

            It is the same test.

            suraj.naik Suraj Naik (Inactive) added a comment - Sergey Avseyev , It is the same test.

            Not seen on 6.5.0-4959

            vikas.chaudhary Vikas Chaudhary added a comment - Not seen on 6.5.0-4959

            People

              satya.nand Satya Nand (Inactive)
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty