Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24509

Moxi truncates documents set with noreply

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 5.0.0
    • 4.6.0
    • moxi
    • Couchbase Server 4.6.0 Centos 6
      Client side Moxi: moxi-server-4.6.0-115.x86_64
      3 Node Vagrant cluster, 4th Vagrant as client
    • Untriaged
    • Unknown

    Description

      When setting a large document (in my testing, around the 20MB limit but also reproducible with less) from a memcached client through Client side Moxi, with the noreply flag set, the document is occasionally truncated.

      couch_dbdump shows the item is in the vbucket file, with the full expected size in the size field:

      [root@node2-cb460-centos6 ~]# /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/default/543.couch.1  | head -c 300
      Dumping "/opt/couchbase/var/lib/couchbase/data/default/543.couch.1":
      Doc seq: 12
           id: 1353
           rev: 12
           content_meta: 131
           size (on disk): 983943
           cas: 1495550766928232448, expiry: 0, flags: 0, datatype: 0
           size: 20971520
           data: (snappy) {"a":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
      

      but, the data is not the full document

      [root@node2-cb460-centos6 ~]# /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/default/543.couch.1  | tail -c 100
      aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
       
      Total docs: 1
      [root@node2-cb460-centos6 ~]# /opt/couchbase/bin/couch_dbdump /opt/couchbase/var/lib/couchbase/data/default/543.couch.1  | wc -c
      134077
      

      Getting this document though a Couchbase client gets the same data as shown by couch_dbdump, but padded with null bytes to the full size of the intended document (causing json parse errors if json is expected).

      It appears there may be a time element to this, as introducing a delay with

      $ tc qdisc add dev eth1 root netem delay 10ms
      

      on the Couchbase node to which this item is being stored (i.e., downstream from the perspective of Moxi) can cause this truncation to appear more often.
      Additionally, enabling verbose III logging with -vvv drastically increases the frequency of this issue.
      The truncation is never seen if noreply is not set.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            james.harrison James Harrison (Inactive)
            james.harrison James Harrison (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty