Details
-
Bug
-
Resolution: Fixed
-
Critical
-
3.2.0, 3.2.2, 3.2.5, 3.2.8
-
None
-
1
Description
Summary
When a .NET application is connected to a cluster with a mix of versions where at least one node supports collections and at least one does not, the mcbp request keys generated by the .NET SDK to non-collections supported nodes sometimes have an extra null byte (0x0) prepended to them.
Such requests are accepted by the Data Service, but the resulting documents are created with the wrong key hence they cannot be accessed by other client applications.
Steps to Reproduce
- Create a cluster consisting of at least 1 node which does not support collections (tested: 6.5.1-MP1), with at least one test bucket.
- Add a second node running 7.0.3.
- Rebalance the cluster so there is a mix of versions.
- Run a .NET program (NCBC_3171_reproducer.zip) which reads and writes keys - for example:
Program.cs
using System;
using System.Threading.Tasks;
using Couchbase;
namespace examples
{
class Program
{
static async Task Main(string[] args)
{
var cluster = await Cluster.ConnectAsync("couchbase://<HOSTNAME>", "Administrator", "asdasd");
var bucket = await cluster.BucketAsync("default");
var collection = bucket.DefaultCollection();
for (;;) {
Console.Write(".");
for (int i = 0; i < 100; i++) {
var upsertResult = await collection.UpsertAsync("key_" + i, new { Name = "Ted", Age = i });
var getResult = await collection.GetAsync("key_" + i);
}
}
}
}
}
Expected behaviour
Keys named "key_0", "key_1"... should be created on the cluster - we can confirm via the UI or by a tool like `cbc-cat`:
Actual behaviour
The document is created with an invalid key - other clients / N1QL cannot access it:
$ /opt/couchbase/bin/cbc-cat -U couchbase://localhost/default -u Administrator -P asdasd key_93
|
key_93 LCB_KEY_ENOENT (0x0D)
|
If we examine the raw in-memory data we can confirm that the document has an invalid key with an extra nul-byte prefixed:
$ /opt/couchbase/bin/cbstats localhost -u Administrator -p asdasd -b default raw "_hash-dump 672" | perl -pe 's/\000/\\u0000/gs'
|
vb:672: HashTable[0x7f1532b8e618] with numItems:1 numInMemory:1 numDeleted:0 numNonResident:0 numTemp:0 numSystemItems:0 numPreparedSW:0 values:
|
SV @0x7f153083dd80 ..J ..R.Cm temp: seq:1 rev:1 cas:1648128940175392768 key:"cid:0x0:\u0000key_93, size:8" exp:0 age:1 nru:1 fc:5 vallen:17 val age:1 :"{"name":"Ted","age":93}"
|
Alternatively the following one-liner can be used to scan an entire node for bad keys (slow on large datasets as it is essentially reading all in-memory data from the HashTables):
for vb in $(seq 0 1023); do echo $vb; (/opt/couchbase/bin/cbstats localhost -u Administrator -p asdasd -b default raw "_hash-dump $vb" | perl -pe 's/\000/\\u0000/gs' | grep \\u0000); done
|
Comments
- Tested with .NET SDK versions 3.2.0, 3.2.2, 3.2.5 and 3.2.8; which all suffer from it.
Not all 6.x nodes appear to be affected: when running with a 3-node cluster where 2 nodes were 6.5.1-MP1 and one was 7.0.3, I only saw invalid keys being generated for one of the 6.5.1-MP3 nodes, not the other.Edit - re-run the same test and I do see corrupt keys on both of the 6.5.1 nodes. Possible I missed them in the first attempt.- Note that once the application is affected by this issue, removing the 7.0.3 node from the cluster does not fix it - it continues to generate incorrect keys.
Attachments
Issue Links
For Gerrit Dashboard: NCBC-3171 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
172926,7 | NCBC-3171: mixed node dev preview CID fix | master | couchbase-net-client | Status: MERGED | +2 | +1 |