Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35208

Rebalance failed during replica update if disk backfill and memory snapshot contain same Prepare key

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-3821

      Scenario: (All doc ops are with durability=MAJORITY)

      1. 4 node cluster, Couchbase bucket (replica=0)
      2. Load 10K docs initially
      3. Update replica=1 then rebalance (Succeeds)
      4. Perform SyncWrite doc update of already existing docs
      5. Updarte replica=2 the rebalance

      Observation:

      Rebalance failed at step-5 with following logs

      Failure logs:

      Rebalance exited with reason {mover_crashed,
      		 
      {unexpected_exit,
      		 
      {'EXIT',<0.13606.18>,
      		 
      {{{{{child_interrupted,
      		 
      {'EXIT',<23958.4765.12>,socket_closed}},
      		 
      [{dcp_replicator,spawn_and_wait,1,
      		 
      [{file,"src/dcp_replicator.erl"}, {line,249}]},
      		 
      {dcp_replicator,handle_call,3,
      		 
      [{file,"src/dcp_replicator.erl"}, {line,121}]},
      		 
      {gen_server,try_handle_call,4,
      		 
      [{file,"gen_server.erl"},{line,636}]},
      		 
      {gen_server,handle_msg,6,
      		 
      [{file,"gen_server.erl"},{line,665}]},
      		 
      {proc_lib,init_p_do_apply,3,
      		 
      [{file,"proc_lib.erl"},{line,247}]}]},
      		 
      {gen_server,call,
      		 
      [<23958.4764.12>,get_partitions, infinity]}},
      		 
      {gen_server,call,
      		 
      ['dcp_replication_manager-default',
      		 
      {get_replicator_pid,750}, infinity]}},
      		 
      {gen_server,call,
      		 
      [{'janitor_agent-default', 'ns_1@10.112.191.102'},
      		 
      {if_rebalance,<0.10265.18>,
      		 
      {update_vbucket_state,496,active, undefined,undefined,
      		 
      [['ns_1@10.112.191.102', 'ns_1@10.112.191.104', 'ns_1@10.112.191.103']]}}, infinity]}}}}}.
      		 
      Rebalance Operation Id = 6cf5d99bc27f13cb6559e84533430f3f 

      TAF testcase:

      bucket_param_tests.bucket_param_update.BucketParamTest.test_replica_update,nodes_init=4,num_items=10000,replicas=0,doc_ops=update;create;delete
      		 
      ;durability=MAJORITY

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-35253 rebalance failed with "dcp_replicator,spawn_and_wait,1" error while loading through transactions with durability as MAJORITY

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-35209 Rebalance Out failed with reason wait_seqno_persisted_failed. Durability=Majority, bucket replica=0

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty