Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55610

[CDC] Rebalance exited with wait_seqno_persisted_failed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 7.2.0
    • couchbase-bucket
    • 7.2.0-5165

    Description

      Steps To Recreate:

      1. Create a 4 node cluster
      2. Create a magma bucket (history_retention_bytes,1000000000, history_retention_seconds,86400)
      3. Create four scopes(including default scope count is 5)
      4. Create five collections in each of the scopes (_default scope has four non default and one default collection)
      5. Create 1 million items in each of the collections
      6. Perform cont. dedupe mutations and trigger rebalance_in for 2 nodes in parallel
      7. Rebalance exited with wait_seqno_persisted_failed

      Rebalance Failure:

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.13757.1>,
      {{wait_seqno_persisted_failed,"bucket-0",673,
      34199,
      [{'ns_1@172.23.107.232',
      {'EXIT',
      {socket_closed,
      {gen_server,call,
      [{'janitor_agent-bucket-0',
      'ns_1@172.23.107.232'},
      {if_rebalance,<0.12475.1>,
      {wait_seqno_persisted,673,34199}},
      infinity]}}}}]},
      [{ns_single_vbucket_mover,
      '-wait_seqno_persisted_many/5-fun-2-',5,
      [{file,"src/ns_single_vbucket_mover.erl"},
      {line,474}]},
      {proc_lib,init_p,3,
      [{file,"proc_lib.erl"},{line,211}]}]}}}}.
      Rebalance Operation Id = ca01b159151c3ded5b22848a0485b675
       
      Worker <0.13514.1> (for action {move,{673,
      ['ns_1@172.23.107.240',
      'ns_1@172.23.107.232',
      'ns_1@172.23.107.97',
      'ns_1@172.23.107.221'],
      ['ns_1@172.23.121.78',
      'ns_1@172.23.107.221',
      'ns_1@172.23.107.232',
      'ns_1@172.23.107.97'],
      []}}) exited with reason {unexpected_exit,
      {'EXIT',
      <0.13757.1>,
      {{wait_seqno_persisted_failed,
      "bucket-0",
      673,34199,
      [{'ns_1@172.23.107.232',
      {'EXIT',
      {socket_closed,
      {gen_server,
      call,
      [{'janitor_agent-bucket-0',
      'ns_1@172.23.107.232'},
      {if_rebalance,
      <0.12475.1>,
      {wait_seqno_persisted,
      673,
      34199}},
      infinity]}}}}]},
      [{ns_single_vbucket_mover,
      '-wait_seqno_persisted_many/5-fun-2-',
      5,
      [{file,
      "src/ns_single_vbucket_mover.erl"},
      {line,
      474}]},
      {proc_lib,
      init_p,3,
      [{file,
      "proc_lib.erl"},
      {line,
      211}]}]}}}
      

      QE-TEST:

      git fetch https://review.couchbase.org/TAF refs/changes/98/186098/16 && git cherry-pick FETCH_HEAD
      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini rerun=False,skip_cluster_reset=True,get-cbcollect-info=False,log_level=debug,upgrade_version=7.2.0-5150 -t bucket_collections.history_retention.DocHistoryRetention.test_rebalance_with_dedupe,nodes_init=4,bucket_spec=single_bucket.history_retention_tests,replicas=3,num_items=1000000,bucket_history_retention_seconds=86400,bucket_history_retention_bytes=1000000000,nodes_in=2,GROUP=P1'
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              vesko.karaganev Vesko Karaganev
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty