Details
-
Bug
-
Resolution: Done
-
Critical
-
6.5.0
-
None
-
Untriaged
-
Unknown
Description
During the kv-engine-jepsen-nightly-341 we observed this error message on node 172.28.128.42:
./20191126T213349.000-0800/172.28.128.42/cbcollect_info_ns_1@172.28.128.42_20191127-053651/memcached.log:2019-11-27T05:36:00.036338+00:00 ERROR 59: (default) DCP (Producer) eq_dcpq:replication:ns_1@172.28.128.42->ns_1@172.28.128.245:default - Disconnecting. Received status Not found for op:DCP_PREPARE response:{"bodylen":0,"cas":0,"datatype":"raw","extlen":0,"keylen":0,"magic":"ClientResponse","opaque":95,"opcode":"DCP_PREPARE","status":"Not found"}
|
While performing test:
lein trampoline run test --nodes-file ./nodes --username vagrant --ssh-private-key ./resources/vagrantkey --workload=rebalance --scenario=sequential-rebalance-out-in --replicas=2 --no-autofailover --disrupt-count=1 --rate=0 --durability=0:100:0:0 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level
Jepsen Test Description:
- Setup 4 node cluster
- Perform read and durability Majority writes (this continues thought out rebalances) over a set of keys treating them as registers
- After 10 seconds rebalance one node out by stating its an ejected node for the rebalance (in this case 172.28.128.246)
- After 10 seconds rebalance the node back into the cluster by adding it to the cluster and then rebalancing it back in (node the add fails in this jepsen run which then means the rebalance fails )
- Wait 5 seconds
- Analysis the linaraziablity of the read and writes this should pass
Attachments
Issue Links
- relates to
-
MB-36971 Rebalance stuck after replica decrement of the bucket.
- Closed