Rebalance in operation fails with error bulk_set_vbucket_state_failed

Build : 5.1.0-1344

A basic testcase for rebalance in of nodes fails with the following error:

Rebalance exited with reason {unexpected_exit, {'EXIT',<0.4839.0>, {bulk_set_vbucket_state_failed, [{'ns_1@172.23.122.25', {'EXIT', {{{badmatch, [{<22041.3486.0>, {done,exit, {{{badmatch,{error,closed, [{dcp_proxy,handle_packet,2, [

{file,"src/dcp_proxy.erl"}, {line,169}]}, {dcp_proxy,process_data_loop,3, [{file,"src/dcp_proxy.erl"}

,

{line,292}]}, {dcp_proxy,handle_info,2, [{file,"src/dcp_proxy.erl"}, {line,98}]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"}, {line,604}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,239}]}]}, {gen_server,call, [<22041.3484.0>, {setup_streams,[1023]}, infinity]}}, [{gen_server,call,3, [{file,"gen_server.erl"}, {line,188}]}, {dcp_replicator, 'spawn_and_wait/1-fun-0',1, [{file,"src/dcp_replicator.erl"}, {line,208}]}]}}]}, [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"},{line,1307}]}, {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}, {line,230}]}, {dcp_replicator,handle_call,3, [{file,"src/dcp_replicator.erl"}, {line,109}]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,585}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]}, {gen_server,call, ['dcp_replicator-default-ns_1@172.23.122.24', {setup_replication,[1023]}, infinity]}}, {gen_server,call, ['replication_manager-default', {change_vbucket_replication,1023, 'ns_1@172.23.122.24'}, infinity]}}, {gen_server,call, [{'janitor_agent-default', 'ns_1@172.23.122.25'}, {if_rebalance,<0.4823.0>, {update_vbucket_state,1023,replica, undefined,'ns_1@172.23.122.24'}}, infinity]}}}}, {'ns_1@172.23.122.27', {'EXIT', {{{badmatch, [{<22033.3410.0>, {done,exit, {{{badmatch,{error,closed, [{dcp_proxy,handle_packet,2, [{file,"src/dcp_proxy.erl"}, {line,169}]}, {dcp_proxy,process_data_loop,3, [{file,"src/dcp_proxy.erl"}, {line,292}

]}, {dcp_proxy,handle_info,2, [

{file,"src/dcp_proxy.erl"}

,

{line,98}

]}, {gen_server,handle_msg,5, [

{file,"gen_server.erl"}, {line,604}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"}, {line,239}]}]}, {gen_server,call, [<22033.3408.0>, {setup_streams,[1023]}, infinity]}}, [{gen_server,call,3, [{file,"gen_server.erl"}

,

{line,188}

]}, {dcp_replicator, 'spawn_and_wait/1-fun-0',1, [

{file,"src/dcp_replicator.erl"}, {line,208}]}]}}]}, [{misc, sync_shutdown_many_i_am_trapping_exits, 1, [{file,"src/misc.erl"},{line,1307}]}, {dcp_replicator,spawn_and_wait,1, [{file,"src/dcp_replicator.erl"}

,

{line,230}

]}, {dcp_replicator,handle_call,3, [

{file,"src/dcp_replicator.erl"}

,

{line,109}

]}, {gen_server,handle_msg,5, [

{file,"gen_server.erl"}

,

{line,585}

]}, {proc_lib,init_p_do_apply,3, [

{file,"proc_lib.erl"}

,

{line,239}

]}]}, {gen_server,call, ['dcp_replicator-default-ns_1@172.23.122.24',

{setup_replication,[1023]}

, infinity]}}, {gen_server,call, ['replication_manager-default',

{change_vbucket_replication,1023, 'ns_1@172.23.122.24'}

, infinity]}}, {gen_server,call, [

{'janitor_agent-default', 'ns_1@172.23.122.27'}

, {if_rebalance,<0.4823.0>, {update_vbucket_state,1023,pending, passive,'ns_1@172.23.122.24'}}, infinity]}}}}]}}}

This test last passed on 5.1.0-1341. It could have regressed due to the following commit :

CHANGELOG for kv_engine

• Commit: df3a25632cedf711aeade4e67096727c8bb957a7 (in build: 1344)
  Author: Trond Norbye
  Refactor: Store the packet in the cookie

The cookie represents a single command so it should contain
the packet data for the actual command. Given that we (currently)
read all of the data into a shared buffer (possibly containing
multiple commands) we don't want to move the ownership of the
data into the cookie (that would mean having to split that
buffer).

Change-Id: I162e9ac6bb078326b45ac111df44efb7face4fcc
Reviewed-on: http://review.couchbase.org/84793
Tested-by: Build Bot
Reviewed-by: Jim Walker
Reviewed-by: Dave Rigby