Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
Cheshire-Cat
-
7.0.0-3938
-
Triaged
-
Centos 64-bit
-
1
-
Unknown
Description
Steps to repro:
- Create a 3 node cluster(replicas=2)
- Rebalance in and out multiple times, during load between rebalance
- Rebalance exited with mover crashed (bulk_set_vbucket_state_failed)
Rebalance Failure:
Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.22590.1>,\n {{bulk_set_vbucket_state_failed,\n [{\'ns_1@172.23.106.63\',\n {\'EXIT\',\n {{{{badmatch,{error,timeout}},\n [{mc_client_binary,cmd_vocal_recv,5,\n [{file,"src/mc_client_binary.erl"},\n {line,155}]},\n {mc_client_binary,set_vbucket,4,\n [{file,"src/mc_client_binary.erl"},\n {line,394}]},\n {ns_memcached,do_handle_call,3,\n [{file,"src/ns_memcached.erl"},\n {line,561}]},\n {ns_memcached,worker_loop,3,\n [{file,"src/ns_memcached.erl"},\n {line,245}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},\n {line,249}]}]},\n {gen_server,call,\n [\'ns_memcached-default\',\n {set_vbucket,546,replica,undefined},\n 180000]}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.106.63\'},\n {if_rebalance,<0.10987.0>,\n {update_vbucket_state,525,replica,\n undefined,undefined}},\n infinity]}}}}]},\n [{janitor_agent,bulk_set_vbucket_state,4,\n [{file,"src/janitor_agent.erl"},\n {line,403}]},\n {ns_single_vbucket_mover,\n \'-cleanup_old_streams/4-fun-1-\',4,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,353}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 7ca0869bfad48cfb72379c71f06c693c'}
|
2020-12-06 05:54:44,838 | test | ERROR | MainThread | [rest_client:print_UI_logs:2596] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.106.60', u'tstamp': 1607262882871L, u'shortText': u'message', u'serverTime': u'2020-12-06T05:54:42.871Z', u'text': u'Worker <0.22380.1> (for action {move,{525,\n [\'ns_1@172.23.106.62\',\n \'ns_1@172.23.106.63\',\n \'ns_1@172.23.106.60\'],\n [\'ns_1@172.23.106.60\',\n \'ns_1@172.23.106.62\',\n \'ns_1@172.23.123.7\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.22590.1>,\n {{bulk_set_vbucket_state_failed,\n [{\'ns_1@172.23.106.63\',\n {\'EXIT\',\n {{{{badmatch,\n {error,\n timeout}},\n [{mc_client_binary,\n cmd_vocal_recv,\n 5,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 155}]},\n {mc_client_binary,\n set_vbucket,\n 4,\n [{file,\n "src/mc_client_binary.erl"},\n {line,\n 394}]},\n {ns_memcached,\n do_handle_call,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 561}]},\n {ns_memcached,\n worker_loop,\n 3,\n [{file,\n "src/ns_memcached.erl"},\n {line,\n 245}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 249}]}]},\n {gen_server,\n call,\n [\'ns_memcached-default\',\n {set_vbucket,\n 546,\n replica,\n undefined},\n 180000]}},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.106.63\'},\n {if_rebalance,\n <0.10987.0>,\n {update_vbucket_state,\n 525,\n replica,\n undefined,\n undefined}},\n infinity]}}}}]},\n [{janitor_agent,\n bulk_set_vbucket_state,\n 4,\n [{file,\n "src/janitor_agent.erl"},\n {line,\n 403}]},\n {ns_single_vbucket_mover,\n \'-cleanup_old_streams/4-fun-1-\',\n 4,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 353}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'}
|
QE TEST:
./testrunner -i /tmp/testexec.44201.ini GROUP=P0;durability,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,bucket_storage=magma,rerun=False,get-cbcollect-info=True,infra_log_level=critical,log_level=error,upgrade_version=7.0.0-3938 -t rebalance_new.rebalance_in_out.RebalanceInOutTests.test_incremental_rebalance_in_out_with_mutation_and_compaction,upgrade_version=7.0.0-3938,rerun=False,GROUP=P0;durability,get-cbcollect-info=True,replicas=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,bucket_storage=magma,log_level=error,nodes_init=3,num_items=400000,infra_log_level=critical
|
JoB:+
http://qa.sc.couchbase.com/job/test_suite_executor-TAF/77019/consoleFull