Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40178

[Magma] Rebalance failed with mover crashed:"badmatch"

    XMLWordPrintable

Details

    Description

      Steps to repro:

      * Created a 3 node cluster(replicas = 1)
      * Loaded 10 million items of 250 bytes each
      * Added 3 new nodes and triggerd rebalance
      * During rebalance new creates and deletions (half of the exisiting elements) was going on
      * Rebalance exited with mover_Crashed  "badmatch"
      

      Rebalance error log:

      Rebalance exited with reason {mover_crashed,
      {unexpected_exit,
      {'EXIT',<0.31095.3>,
      {{{{badmatch,{error,closed}},
      [{mc_client_binary,cmd_vocal_recv,5,
      [{file,"src/mc_client_binary.erl"},
      {line,155}]},
      {mc_client_binary,set_cluster_config,4,
      [{file,"src/mc_client_binary.erl"},
      {line,743}]},
      {ns_memcached,do_handle_call,3,
      [{file,"src/ns_memcached.erl"},
      {line,559}]},
      {ns_memcached,worker_loop,3,
      [{file,"src/ns_memcached.erl"},
      {line,236}]},
      {proc_lib,init_p_do_apply,3,
      [{file,"proc_lib.erl"},{line,247}]}]},
      {gen_server,call,
      ['ns_memcached-default',
      {get_dcp_docs_estimate,127,
      "replication:ns_1@172.23.123.8->ns_1@172.23.123.3:default"},
      180000]}},
      {gen_server,call,
      [{'janitor_agent-default',
      'ns_1@172.23.123.8'},
      {if_rebalance,<0.26959.3>,
      {wait_dcp_data_move,
      ['ns_1@172.23.123.3','ns_1@172.23.123.9'],
      127}},
      infinity]}}}}}.
      Rebalance Operation Id = 3e4ccf2c5cfc66b201a78137a96624b2 hide
      

      QE test:

      ./testrunner -i /tmp/ankush.ini  -t rebalance_new.rebalance_in.RebalanceInTests.test_rebalance_in_with_ops,nodes_init=3,nodes_in=3,replicas=1,num_items=10000000,bucket_storage=magma,bucket_eviction_policy=fullEviction,randomize_value=True,vbuckets=128,rerun=false,get-cbcollect-info=True,upgrade_version=7.0.0-2422,skip_cleanup=False
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ankush.sharma Ankush Sharma
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty