Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39503

Receiving interlaced vBucket mutations when using sequential backfill

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 6.6.0
    • couchbase-bucket
    • None

    Description

      What's the problem?
      When setting the new 'backfill_order' control flag to 'sequential' as expected I'm seeing the majority of mutations in sequential vBucket order, however, during the switch from one vBucket to the next I'm receiving interlaced mutations.

      What am I expecting to see?
      If I was stream two vBuckets, I would expected to see all mutations and the stream end for the first vBucket before receiving any mutations for the next vBucket.

      Steps to reproduce
      1) Use the TLM with the 'mad-hatter' manifest to sync all the required repositories
      2) Change directory to '$SOURCE/godeps/src/github.com/couchbase/gocbcore/v9' and apply this patch to enable the control flag in gocbcore.
      3) Change directory to '$SOURCE/goproj/src/github.com/couchbase/backup' and apply this patch to disable synchronous vBucket streaming and to use the new 'backfill_order' control flag.
      4) Rebuild
      5) Spin up a new Couchbase Server node
      6) To enable reproducing with less data change the number of vBuckets from 64/1024 to 16 with 'curl -X POST -u Administrator:password http://$SERVER_IP/diag/eval -d "ns_config:set(couchbase_num_vbuckets_default, 16)."'.
      7) Load 10,000 1KB items using cbbackupmgr/cbworkloadgen
      8) Configure an archive using 'cbbackupmgr config -a $ARCHIVE -r $REPO'
      9) Perform a backup using a single thread (meaning we should only create a single DCP worker/connection) with 'cbbackupmgr backup -a $ARCHIVE -r $REPO -c $SERVER_IP -u Administrator -p password --threads 1'.

      If a packet capture is taken with 'tcpdump -i any -C 1024 -w $FILENAME -s 0 port 11210' we should see interlaced mutations. I've attached a packet capture which contains interlaced mutations between packets 374-1087 when filtered using 'couchbase.opcode == 0x57' in Wireshark.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              james.lee James Lee
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty