Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.6.0
-
Couchbase Server 4.6.0 Centos 6
Client side Moxi: moxi-server-4.6.0-115.x86_64
3 Node Vagrant cluster, 4th Vagrant as client
-
Untriaged
-
Unknown
Description
When setting a large document (in my testing, around the 20MB limit but also reproducible with less) from a memcached client through Client side Moxi, with the noreply flag set, the document is occasionally truncated.
couch_dbdump shows the item is in the vbucket file, with the full expected size in the size field:
[root@node2-cb460-centos6 ~]# /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/default/543.couch.1 | head -c 300
|
Dumping "/opt/couchbase/var/lib/couchbase/data/default/543.couch.1":
|
Doc seq: 12
|
id: 1353
|
rev: 12
|
content_meta: 131
|
size (on disk): 983943
|
cas: 1495550766928232448, expiry: 0, flags: 0, datatype: 0
|
size: 20971520
|
data: (snappy) {"a":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
|
but, the data is not the full document
[root@node2-cb460-centos6 ~]# /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/default/543.couch.1 | tail -c 100
|
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
|
|
Total docs: 1
|
[root@node2-cb460-centos6 ~]# /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/default/543.couch.1 | wc -c
|
134077
|
Getting this document though a Couchbase client gets the same data as shown by couch_dbdump, but padded with null bytes to the full size of the intended document (causing json parse errors if json is expected).
It appears there may be a time element to this, as introducing a delay with
$ tc qdisc add dev eth1 root netem delay 10ms
|
on the Couchbase node to which this item is being stored (i.e., downstream from the perspective of Moxi) can cause this truncation to appear more often.
Additionally, enabling verbose III logging with -vvv drastically increases the frequency of this issue.
The truncation is never seen if noreply is not set.
Attachments
For Gerrit Dashboard: MB-24509 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
78763,5 | MB-24509: Prevent truncation of documents set with noreply | master | moxi | Status: MERGED | +2 | +1 |