Rebalance failed: Packet validation failed for "DCP_DELETION" - Failed to inflate payload

+----------------+----------+--------------+

| Nodes          | Services | Status       |

+----------------+----------+--------------+

| 172.23.120.29  | kv       | Cluster node |

| 172.23.120.255 | [u'kv']  | --- OUT ---> |

| 172.23.121.25  | kv       | Cluster node |

+----------------+----------+--------------+

Rebalance exited with reason {mover_crashed,

{unexpected_exit,'EXIT',<0.4969.0>,

{{dcp_wait_for_data_move_failed,"default",338,

'ns_1@172.23.120.255',['ns_1@172.23.121.25'],{error,no_stats_for_this_vbucket}},

[{ns_single_vbucket_mover,'-wait_dcp_data_move/5-fun-0-',5,[{file,"src/ns_single_vbucket_mover.erl"},{line,465}]},

{proc_lib,init_p,3,[{file,"proc_lib.erl"},{line,234}]}]}}}}.Operation Id = f979e3f27acecff3b0d4b1b952f971f7'}

'Worker <0.4893.0> (

for action {move,{338,['ns_1@172.23.120.255'],['ns_1@172.23.121.25'],[]}}) exited with reason

{unexpected_exit,{'EXIT',<0.4969.0>,

{{dcp_wait_for_data_move_failed,"default",338,'ns_1@172.23.120.255',['ns_1@172.23.121.25'],{error,no_stats_for_this_vbucket}},

 [{ns_single_vbucket_mover,'-wait_dcp_data_move/5-fun-0-',5,[{file,"src/ns_single_vbucket_mover.erl"},{line,465}]},

 {proc_lib,init_p,3,[{file,"proc_lib.erl"},{line,234}]}]}}}'}

172.23.120.255: Found 'ERROR' logs :: 

2020-12-14T17:27:05.358539-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358598-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358614-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358630-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358645-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358667-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358824-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358881-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358971-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.358986-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359007-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359021-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359038-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359137-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359158-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.359184-08:00 ERROR 110: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":1,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.120.29:default, vb:170, state:in-memory\n', '2020-12-14T17:27:05.373581-08:00 ERROR 111: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.121.25:default - DcpProducer::handleResponse disconnecting, received unexpected response:{"bodylen":49,"cas":0,"datatype":["JSON"],"extlen":0,"keylen":0,"magic":"ClientResponse","opaque":18,"opcode":"DCP_DELETION","status":"Invalid arguments"} for stream:stream name:eq_dcpq:replication:ns_1@172.23.120.255->ns_1@172.23.121.25:default, vb:338, state:in-memory

rebalance_new.rebalance_out.RebalanceOutTests:

    rebalance_out_with_ops,nodes_init=3,nodes_out=1,replicas=0,num_items=20000,doc_ops=create:update:delete,durability=MAJORITY,sdk_client_pool=True,sdk_timeout=60