Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-20746

get_meta_on_set_meta ops and false positive disk fetches happen if the destination bucket is compacted prior to XDCR replication

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Steps:
      1) Create two single-node clusters ("source" and "destination").
      2) Create a bucket on source and destination nodes.
      3) Insert 1M docs into the source node (actual number of documents probably doesn't matter).
      4) Tune XDCR settings (all documents will be replicated optimistically):

      optimisticReplicationThreshold=10240
      targetNozzlePerNode=64
      sourceNozzlePerNode=64
      

      5) Enable uni-directional replication from the source to the destination.

      All stats look OK:

      /opt/couchbase/bin/cbstats -b bucket-1 -p password 127.0.0.1:11210 all | grep "ep_num_ops_get_meta_on_set_meta\|ep_bg_meta_fetched" 
       ep_bg_meta_fetched:                                    0
       ep_num_ops_get_meta_on_set_meta:                       0
      

      However, if the destination bucket is compacted prior to the step 5, then stats are completely different:

      /opt/couchbase/bin/cbstats  -b bucket-1 -p password 127.0.0.1:11210 all | grep "ep_num_ops_get_meta_on_set_meta\|ep_bg_meta_fetched" 
       ep_bg_meta_fetched:                                    1000000
       ep_num_ops_get_meta_on_set_meta:                       1000000
      

      Unfortunately, it's not a stats issue. There are actual fetches that have a negative impact on XDCR performance:

       bg_wait (1000000 total)
          1us - 2us     : (  0.00%)     42 
          2us - 4us     : (  0.10%)   1008 
          4us - 8us     : (  0.59%)   4850 
          8us - 16us    : (  5.64%)  50458 ##
          16us - 32us   : ( 12.97%)  73295 ###
          32us - 64us   : ( 21.36%)  83984 ###
          64us - 128us  : ( 33.15%) 117827 ####
          128us - 256us : ( 48.20%) 150575 ######
          256us - 512us : ( 65.06%) 168570 ######
          512us - 1ms   : ( 82.31%) 172469 #######
          1ms - 2ms     : ( 96.08%) 137766 #####
          2ms - 4ms     : ( 99.80%)  37151 #
          4ms - 8ms     : ( 99.99%)   1890 
          8ms - 16ms    : (100.00%)    115 
          Avg           : (  388us)
      

      Attachments

        1. dest.zip
          22.84 MB
        2. source.zip
          26.27 MB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sriram Sriram Ganesan (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty