Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60846

[Rebalance] : Rebalance exited with reason {mover_crashed,{unexpected_exit,{'EXIT',<0.13829.9>,{noproc,{gen_server,call,[{'janitor_agent-default','ns_1@ec2-3-80-24-14.compute-1.amazonaws.com'},{if_rebalance,<0.19451.8>,{wait_dcp_data_move

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. Created a 1 node kv cluster 
      2. Created a couchstore bucket named "default"
      3. Three views default_view0, default_view1, default_view2 was created successfully in ddoc: ddoc1
      4. A lot of view queries were running continuously throughout the test
      5. Added 2 more nodes and started a rebalance - Rebalance succeeds
      6. Added 3 more nodes and started a rebalance - Rebalance fails

       

      2024-02-18T06:36:16.098Z, ns_vbucket_mover:0:critical:message(ns_1@ec2-54-167-90-65.compute-1.amazonaws.com) - Worker <0.13594.9> (for action {move,{100,                                      ['ns_1@ec2-3-80-24-14.compute-1.amazonaws.com',                                       'ns_1@ec2-44-220-95-241.compute-1.amazonaws.com'],                                      ['ns_1@ec2-44-220-95-241.compute-1.amazonaws.com',                                       'ns_1@ec2-107-21-70-122.compute-1.amazonaws.com'],                                      []}}) exited with reason {unexpected_exit,                                                                {'EXIT',                                                                 <0.13829.9>,                                                                 {noproc,                                                                  {gen_server,                                                                   call,                                                                   [{'janitor_agent-default',                                                                     'ns_1@ec2-3-80-24-14.compute-1.amazonaws.com'},                                                                    {if_rebalance,                                                                     <0.19451.8>,                                                                     {wait_dcp_data_move,                                                                      ['ns_1@ec2-44-220-95-241.compute-1.amazonaws.com',                                                                       'ns_1@ec2-107-21-70-122.compute-1.amazonaws.com'],                                                                      100}},                                                                    infinity]}}}}2024-02-18T06:36:16.141Z, ns_orchestrator:0:critical:message(ns_1@ec2-54-167-90-65.compute-1.amazonaws.com) - Rebalance exited with reason {mover_crashed,                              {unexpected_exit,                               {'EXIT',<0.13829.9>,                                {noproc,                                 {gen_server,call,                                  [{'janitor_agent-default',                                    'ns_1@ec2-3-80-24-14.compute-1.amazonaws.com'},                                   {if_rebalance,<0.19451.8>,                                    {wait_dcp_data_move,                                     ['ns_1@ec2-44-220-95-241.compute-1.amazonaws.com',                                      'ns_1@ec2-107-21-70-122.compute-1.amazonaws.com'],                                     100}},                                   infinity]}}}}}.Rebalance Operation Id = 0a0f1f466b3c09cf13d0677d059163b7

       

       

       

       

      Testrunner script to reproduce

      ./testrunner -i /data/workspace/alma9-p0-os_certify-vset00-00-view/testexec.84921.ini -p get-delays=true,get-cbcollect-info=False,get-cbcollect-info=True,get-cbcollect-info=True,get-cbcollect-info=True,hostname=true,sirius_url=http://172.23.120.103:4000 -t rebalance.rebalancein.RebalanceInTests.incremental_rebalance_in_with_queries,bucket_storage=couchstore,blob_generator=False,items=2000,is_dev_ddoc=False,max_verify=2000,GROUP=P1

      Job name : alma9-os-certify-view

      Job ref : http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-2149/jenkins_logs/test_suite_executor/680972/

       

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-60876 [BP of MB-60846 to 7.2.5] : Rebalance exited with reason {mover_crashed,{unexpected_exit,{'EXIT',<0.13829.9>,{noproc,{gen_server,call,[{'janitor_agent-default','ns_1@ec2-3-80-24-14.compute-1.amazonaws.com'},{if_rebalance,<0.19451.8>,{wait_dcp_data_move

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-60682 Rebalance exited with reason {mover_crashed,{unexpected_exit,{'EXIT',<0.20719.9>,{{bulk_set_vbucket_state_failed,[{'ns_1@ec2-3-86-84-232.compute-1.amazonaws.com',{'EXIT',{{error,{badrpc,{'EXIT',{{badarg,[{ets,lookup,[set_view_by_ddoc_id_ets

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty