Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43017

[Magma] Rebalance exited with mover crashed {{{{nocatch,{error,timeout}}

    XMLWordPrintable

Details

    Description

      Steps to repro:

      1. Create a 3 node cluster(replicas=1)
      2. Created 3 buckets .
      3. Created 3 collecitons in each bucket
      4. Created 10 documents in each collection(total 30 in each bucket, doc-size=200b)
      5. Started doc ops(Create:update:delete:expiry)
      6. Deleted one collection in each of the bucket
      7. During doc ops "Triggered rebalance in"
      8. Rebalance exited with mover crashed

      Rebalance failure:

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.15098.386>,
      {{{{nocatch,{error,timeout}},
      [{mc_binary,recv_with_data,4,
      [{file,"src/mc_binary.erl"},{line,48}]},
      {mc_binary,quick_stats_recv,3,
      [{file,"src/mc_binary.erl"},{line,53}]},
      {mc_binary,quick_stats_loop_enter,5,
      [{file,"src/mc_binary.erl"},{line,105}]},
      {mc_binary,quick_stats,5,
      [{file,"src/mc_binary.erl"},{line,90}]},
      {ns_memcached,do_handle_call,3,
      [{file,"src/ns_memcached.erl"},
      {line,572}]},
      {ns_memcached,worker_loop,3,
      [{file,"src/ns_memcached.erl"},
      {line,245}]},
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"},{line,249}]}]},
      {gen_server,call,
      ['ns_memcached-magma-1',
      {get_dcp_docs_estimate,952,
      "replication:ns_1@172.23.120.63->ns_1@172.23.120.64:magma-1"},
      180000]}},
      {gen_server,call,
      [{'janitor_agent-magma-1',
      'ns_1@172.23.120.63'},
      {if_rebalance,<0.21799.385>,
      {wait_dcp_data_move,
      ['ns_1@172.23.120.64',
      'ns_1@172.23.120.65'],
      952}},
      infinity]}}}}}.
      Rebalance Operation Id = 515627b015d2f63ead9cf4e39cde106d 
      

      QE TEST:

      ./testrunner -i /tmp/ankusht.ini  -t magma.magma_rebalance.MagmaRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,rerun=false,sdk_timeout=60,get-cbcollect-info=False,bucket_storage=magma,vbuckets=1024,bucket_eviction_policy=fullEviction,skip_cleanup=True,sdk_timeout=60,standard_buckets=3,infra_log_level=debug,log_level=debug,magma_buckets=3,doc_ops=create:update:expiry:delete,num_items=10,doc_size=200,num_collections=3,skip_validations=False,deep_copy=True,data_load_stage=before,num_collections_to_drop=1
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            owend Daniel Owen
            ankush.sharma Ankush Sharma
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty