Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39503

Receiving interlaced vBucket mutations when using sequential backfill

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not a Bug
    • 6.6.0
    • None
    • couchbase-bucket
    • None

    Description

      What's the problem?
      When setting the new 'backfill_order' control flag to 'sequential' as expected I'm seeing the majority of mutations in sequential vBucket order, however, during the switch from one vBucket to the next I'm receiving interlaced mutations.

      What am I expecting to see?
      If I was stream two vBuckets, I would expected to see all mutations and the stream end for the first vBucket before receiving any mutations for the next vBucket.

      Steps to reproduce
      1) Use the TLM with the 'mad-hatter' manifest to sync all the required repositories
      2) Change directory to '$SOURCE/godeps/src/github.com/couchbase/gocbcore/v9' and apply this patch to enable the control flag in gocbcore.
      3) Change directory to '$SOURCE/goproj/src/github.com/couchbase/backup' and apply this patch to disable synchronous vBucket streaming and to use the new 'backfill_order' control flag.
      4) Rebuild
      5) Spin up a new Couchbase Server node
      6) To enable reproducing with less data change the number of vBuckets from 64/1024 to 16 with 'curl -X POST -u Administrator:password http://$SERVER_IP/diag/eval -d "ns_config:set(couchbase_num_vbuckets_default, 16)."'.
      7) Load 10,000 1KB items using cbbackupmgr/cbworkloadgen
      8) Configure an archive using 'cbbackupmgr config -a $ARCHIVE -r $REPO'
      9) Perform a backup using a single thread (meaning we should only create a single DCP worker/connection) with 'cbbackupmgr backup -a $ARCHIVE -r $REPO -c $SERVER_IP -u Administrator -p password --threads 1'.

      If a packet capture is taken with 'tcpdump -i any -C 1024 -w $FILENAME -s 0 port 11210' we should see interlaced mutations. I've attached a packet capture which contains interlaced mutations between packets 374-1087 when filtered using 'couchbase.opcode == 0x57' in Wireshark.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            drigby Dave Rigby added a comment -

            6) To enable reproducing with less data change the number of vBuckets from 64/1024 to 16 with 'curl -X POST -u Administrator:password http://$SERVER_IP/diag/eval -d "ns_config:set(couchbase_num_vbuckets_default, 16)."'.

            FYI you can also achieve this by setting the env var COUCHBASE_NUM_VBUCKETS=N before starting the server - e.g.

            COUCHBASE_NUM_VBUCKETS=16 ./cluster_run --nodes=1
            

            drigby Dave Rigby added a comment - 6) To enable reproducing with less data change the number of vBuckets from 64/1024 to 16 with 'curl -X POST -u Administrator:password http://$SERVER_IP/diag/eval -d "ns_config:set(couchbase_num_vbuckets_default, 16)."'. FYI you can also achieve this by setting the env var COUCHBASE_NUM_VBUCKETS =N before starting the server - e.g. COUCHBASE_NUM_VBUCKETS=16 ./cluster_run --nodes=1
            drigby Dave Rigby added a comment -

            TL;DR: This is expected behaviour given how KV-Engine manages the various DCP queues to send data to clients. The interleaving should be significantly reduced (affecting only a small percentage of mutations when the "end" of one vbucket backfill is reached and the start of the next backfill begins.

            What is happening here is the actual backfills from disk do happen in sequential order, however they might not get put onto the wire in “perfect” order.

            This is because each Stream (Vbucket) has it’s own ready queue which items are pushed into (from backfill, or in-memory phase if you’re using that), and then the Producer (overall DCP connection) takes items off the ready queues in roughly “fair” order.

            What you are seeing is that when one backfill finishes (e.g. vb:0), the background NonIO thread will immediately move onto iterating over the next (e.g. vb:1). If the DCP producer hasn’t drained all items from vb:0's ready queue before disk returns with data from vb:1, you’ll have a situation where both vb:0 and vb:1 have data in their ready queue. Which one is selected and pushed over the network is essentially arbitrary.

            In the above description, the interleaving was exacerbated due to the small data size. Note there’s only allowed to be 20MB of backfill data pending, so say you have only 2MB of data per vBucket, that could mean that there’s 10 vBucket’s worth of backfilled data ready to be sent out (and you could see mutations from 10 vBuckets at the “same” time).

            However, in a real world dataset where each vBucket is > 20MB, then I think you’ll only see a “crossover” of 2 vBuckets. Certainly when I tested locally with 64MB vBuckets (running with 16 vbuckets and 1GB data) I’ve only observed occasional “crossover” - and only for two adjacent vBuckets- by the time you get any data backfilled for say vb:2, all of vb:0’s has been sent.

            I think the crossover is pretty minor in real-world datasets, and to “fix” it to be always perfectly in order would be expensive, both in terms of development (we’d have to serialize stuff which is currently parallelised) and performance (we’d loose some parallelism / possibely have to wait for one vBucket to stream out before spinning up the BG thread to start reading the next).

            As such, resolving this as "Not a Bug".

            drigby Dave Rigby added a comment - TL;DR: This is expected behaviour given how KV-Engine manages the various DCP queues to send data to clients. The interleaving should be significantly reduced (affecting only a small percentage of mutations when the "end" of one vbucket backfill is reached and the start of the next backfill begins. — What is happening here is the actual backfills from disk do happen in sequential order, however they might not get put onto the wire in “perfect” order. This is because each Stream (Vbucket) has it’s own ready queue which items are pushed into (from backfill, or in-memory phase if you’re using that), and then the Producer (overall DCP connection) takes items off the ready queues in roughly “fair” order. What you are seeing is that when one backfill finishes (e.g. vb:0), the background NonIO thread will immediately move onto iterating over the next (e.g. vb:1). If the DCP producer hasn’t drained all items from vb:0's ready queue before disk returns with data from vb:1, you’ll have a situation where both vb:0 and vb:1 have data in their ready queue. Which one is selected and pushed over the network is essentially arbitrary. In the above description, the interleaving was exacerbated due to the small data size. Note there’s only allowed to be 20MB of backfill data pending, so say you have only 2MB of data per vBucket, that could mean that there’s 10 vBucket’s worth of backfilled data ready to be sent out (and you could see mutations from 10 vBuckets at the “same” time). However, in a real world dataset where each vBucket is > 20MB, then I think you’ll only see a “crossover” of 2 vBuckets. Certainly when I tested locally with 64MB vBuckets (running with 16 vbuckets and 1GB data) I’ve only observed occasional “crossover” - and only for two adjacent vBuckets- by the time you get any data backfilled for say vb:2, all of vb:0’s has been sent. I think the crossover is pretty minor in real-world datasets, and to “fix” it to be always perfectly in order would be expensive, both in terms of development (we’d have to serialize stuff which is currently parallelised) and performance (we’d loose some parallelism / possibely have to wait for one vBucket to stream out before spinning up the BG thread to start reading the next). As such, resolving this as "Not a Bug".

            People

              owend Daniel Owen
              james.lee James Lee
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty