Details
-
Bug
-
Resolution: Incomplete
-
Major
-
Cheshire-Cat
-
7.0.0-3874
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Steps to repro:
- Create a 3 node cluster(replicas=1)
- Created 3 buckets .
- Created 3 collecitons in each bucket
- Created 10 documents in each collection(total 30 in each bucket, doc-size=200b)
- Started doc ops(Create:update:delete:expiry)
- Deleted one collection in each of the bucket
- During doc ops "Triggered rebalance in"
- Rebalance exited with mover crashed
Rebalance failure:
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.15098.386>,
|
{{{{nocatch,{error,timeout}},
|
[{mc_binary,recv_with_data,4,
|
[{file,"src/mc_binary.erl"},{line,48}]},
|
{mc_binary,quick_stats_recv,3,
|
[{file,"src/mc_binary.erl"},{line,53}]},
|
{mc_binary,quick_stats_loop_enter,5,
|
[{file,"src/mc_binary.erl"},{line,105}]},
|
{mc_binary,quick_stats,5,
|
[{file,"src/mc_binary.erl"},{line,90}]},
|
{ns_memcached,do_handle_call,3,
|
[{file,"src/ns_memcached.erl"},
|
{line,572}]},
|
{ns_memcached,worker_loop,3,
|
[{file,"src/ns_memcached.erl"},
|
{line,245}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,249}]}]},
|
{gen_server,call,
|
['ns_memcached-magma-1',
|
{get_dcp_docs_estimate,952,
|
"replication:ns_1@172.23.120.63->ns_1@172.23.120.64:magma-1"},
|
180000]}},
|
{gen_server,call,
|
[{'janitor_agent-magma-1',
|
'ns_1@172.23.120.63'},
|
{if_rebalance,<0.21799.385>,
|
{wait_dcp_data_move,
|
['ns_1@172.23.120.64',
|
'ns_1@172.23.120.65'],
|
952}},
|
infinity]}}}}}.
|
Rebalance Operation Id = 515627b015d2f63ead9cf4e39cde106dÂ
|
QE TEST:
./testrunner -i /tmp/ankusht.ini -t magma.magma_rebalance.MagmaRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,rerun=false,sdk_timeout=60,get-cbcollect-info=False,bucket_storage=magma,vbuckets=1024,bucket_eviction_policy=fullEviction,skip_cleanup=True,sdk_timeout=60,standard_buckets=3,infra_log_level=debug,log_level=debug,magma_buckets=3,doc_ops=create:update:expiry:delete,num_items=10,doc_size=200,num_collections=3,skip_validations=False,deep_copy=True,data_load_stage=before,num_collections_to_drop=1
|