Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34537

[Durability] Rebalance failed with "mover crashed" in bucket_replica update after sync_writes

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-3465

      Scenario:

      1. Create 4 node cluster
      2. Create "default" CB bucket (replica=0)
      3. Load 10K docs into bucket with durability=MAJORITY (Succeeded)
      4. Update replica of the bucket to '1'
      5. Start rebalance

      Observation:

      Rebalance failed as soon it starts with following prints,

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.4124.0>,
      {{bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.205',
      {'EXIT', {{{{{child_interrupted,
      {'EXIT',<17583.3130.0>, socket_closed}},
      [{dcp_replicator,spawn_and_wait,1,
      [{file,"src/dcp_replicator.erl"}, {line,249}]},
      {dcp_replicator,handle_call,3,
      [{file,"src/dcp_replicator.erl"}, {line,121}]},
      {gen_server,try_handle_call,4,
      [{file,"gen_server.erl"}, {line,636}]},
      {gen_server,handle_msg,6,
      [{file,"gen_server.erl"}, {line,665}]},
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"}, {line,247}]}]},
      {gen_server,call,
      [<17583.3129.0>,get_partitions, infinity]}},
      {gen_server,call,
      ['dcp_replication_manager-default',
      {get_replicator_pid,1020}, infinity]}},
      {gen_server,call,
      [{'janitor_agent-default', 'ns_1@172.23.105.205'},
      {if_rebalance,<0.3884.0>,
      {update_vbucket_state,1021,replica,
      undefined,'ns_1@172.23.105.206'}}, infinity]}}}}]},
      [{janitor_agent,bulk_set_vbucket_state,4,
      [{file,"src/janitor_agent.erl"}, {line,426}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,232}]}]}}}}.
      Operation Id = da6cfd36c1a0f3004e26f98ff1a4386c
       
      <0.3979.0> exited with {unexpected_exit,
      {'EXIT',<0.4124.0>,
      {{bulk_set_vbucket_state_failed,
      [{'ns_1@172.23.105.205',
      {'EXIT',
      {{{{{child_interrupted,
      {'EXIT',<17583.3130.0>,socket_closed}},
      [{dcp_replicator,spawn_and_wait,1,
      [{file,"src/dcp_replicator.erl"}, {line,249}]},
      {dcp_replicator,handle_call,3,
      [{file,"src/dcp_replicator.erl"}, {line,121}]},
      {gen_server,try_handle_call,4,
      [{file,"gen_server.erl"},{line,636}]},
      {gen_server,handle_msg,6,
      [{file,"gen_server.erl"},{line,665}]},
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"},{line,247}]}]},
      {gen_server,call,
      [<17583.3129.0>,get_partitions,infinity]}},
      {gen_server,call,
      ['dcp_replication_manager-default',
      {get_replicator_pid,1020}, infinity]}},
      {gen_server,call,
      [{'janitor_agent-default',
      'ns_1@172.23.105.205'},
      {if_rebalance,<0.3884.0>,
      {update_vbucket_state,1021,replica,
      undefined,'ns_1@172.23.105.206'}}, infinity]}}}}]},
      [{janitor_agent,bulk_set_vbucket_state,4,
      [{file,"src/janitor_agent.erl"},{line,426}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,232}]}]}}}

      UI log print:

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty