Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44294

[Couchstore]: Rebalance failed due to mover crashed in get_dcp_docs_estimate failed.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • qe
    • 7.0.0-4432

    Description

      Note: This test was intended to run for 5-6 days but failed in 24 hours

      Steps:

      1. Create a 15 node cluster
      2. Create required buckets and collections.
      3. Create 100000000 items sequentially
      4. Rebalance IN with Loading of docs
      5. Rebalance OUT with Loading of docs
      6. Rebalance SWAP with Loading of docs
      7. Rebalance IN/OUT with Loading of docs
      8. Rebalance OUT/IN with Loading of docs
      9. Rebalance IN with Loading of docs
      10. Rebalance OUT with Loading of docs
      11. Rebalance SWAP with Loading of docs
      12. Rebalance IN/OUT with Loading of docs
      13. Rebalance OUT/IN with Loading of docs
      14. Rebalance IN with Loading of docs
      15. Rebalance OUT with Loading of docs
      16. Rebalance SWAP with Loading of docs
      17. Rebalance IN/OUT with Loading of docs
      18. Rebalance OUT/IN with Loading of docs
      19. Rebalance IN with Loading of docs
      20. Rebalance OUT with Loading of docs
      21. Rebalance SWAP with Loading of docs
      22. Rebalance IN/OUT with Loading of docs
      23. Rebalance OUT/IN with Loading of docs
      24. Rebalance IN with Loading of docs

      Rebalance exited with reason {mover_crashed,
      		 
      {unexpected_exit,
      		 
      {'EXIT',<0.8159.423>,
      		 
      {{{{compromised_reply,{error,timeout}},
      		 
      [{ns_memcached,worker_loop,3,
      		 
      [{file,"src/ns_memcached.erl"},
      		 
      {line,233}]},
      		 
      {proc_lib,init_p_do_apply,3,
      		 
      [{file,"proc_lib.erl"},{line,249}]}]},
      		 
      {gen_server,call,
      		 
      ['ns_memcached-GleamBookUsers0',
      		 
      {get_dcp_docs_estimate,353,
      		 
      "replication:ns_1@172.23.121.129->ns_1@172.23.121.116:GleamBookUsers0"},
      		 
      180000]}},
      		 
      {gen_server,call,
      		 
      [{'janitor_agent-GleamBookUsers0',
      		 
      'ns_1@172.23.121.129'},
      		 
      {if_rebalance,<0.20838.393>,
      		 
      {wait_dcp_data_move,
      		 
      ['ns_1@172.23.120.170',
      		 
      'ns_1@172.23.121.116'],
      		 
      353}},
      		 
      infinity]}}}}}.
      		 
      Rebalance Operation Id = 4e3ca836c5e2089de5d6ef78519b8dd4

      Test Details:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/test_job_magma.ini -p bucket_storage=couchstore,bucket_eviction_policy=fullEviction,rerun=False -t volumetests.Magma.volume.test_long_rebalance,nodes_init=15,replicas=1,skip_cleanup=True,num_items=500000,num_buckets=1,bucket_names=GleamBook,doc_size=256,bucket_type=membase,compression_mode=off,iterations=10,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=info,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,maxttl=300,num_buckets=2,num_scopes=10,num_collections=5,doc_ops=expiry,durability=MAJORITY,pc=1,sdk_client_pool=True,get-cbcollect-info=True -m rest'

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty