- Create a single node cluster of Couchbase Server 5.0.1
- Add a Couchbase Server 6.5.0 node and rebalance
- Create a bucket, my example uses test
- Create a document that resides on the 6.5.0 node, this command works if you followed my steps :
- Delete the document that was just created:
- Add a new 6.5.0 node and mark the other 6.5.0 node for removal (so that you guarantee the document will move from one 6.5.x node to the other).
Rebalance fails due to the error:
All subsequent attempts to rebalance all fail.
This is because DCP deletion packets between the two 6.5.0 nodes are being incorrectly handled:
This then closes the DCP connection and ultimately leads to the rebalance failure.
The invalid DCP deletion packet is for the following doc:
DELETE_TIMES has not been negotiated as part of the DCP stream:
This is not negotiated because there is still a 5.0.x node in the cluster that is not DELETE_TIMES aware, however from a packet capture (MB-38390.pcap) it seems that the 6.5 nodes are sending DCP_DELETION_V2 packets, despite it not being negotiated:
This is obviously rejected by the consumer as it is expected an extras length of 18, rather than 21, as covered in https://github.com/couchbase/kv_engine/blob/master/docs/dcp/documentation/commands/deletion.md.
Note: Haven't attached cbcollect_infos as the issue is readily reproducible 100% of the time by following the steps, if needed I can go ahead and generate these though.
This has the net effect that for any cluster where there are deleted documents, it is impossible to directly upgrade from Couchbase Server <5.5.0 to Couchbase Server 6.5.0.
The workaround is to upgrade to an intermediate version before moving to 6.5.0, such as Couchbase Server 6.0.4, which through testing does not show the same issue.
|For Gerrit Dashboard: MB-38390|
|124159,4||MB-38390: Make enable_expiry_opcode respect includeDeleteTime||mad-hatter||kv_engine||Status: MERGED||+2||+1|
|124518,1||Merge remote-tracking branch 'couchbase/mad-hatter' into 'couchbase/master'||master||kv_engine||Status: MERGED||+2||+1|