Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6576

Seeing very frequent compaction on clusters, during xdcr

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Incomplete
    • Affects Version/s: 2.0-beta, 2.0
    • Fix Version/s: 2.0
    • Component/s: XDCR
    • Security Level: Public
    • Labels:
      None
    • Environment:
      2.0-1700
      xdcr, unidirectional replication.
      4g, 4 core machines
      1024 vbuckets

      Description

      There is almost continuous compaction happening on source/ destination clusters during xdcr replication.

      -Loaded 2M data on source, this data is being replicated to the destination cluster.
      -Compaction setting is set to 30% database fragmenation [default setting]

      Not sure why it is triggered atleast 5-6 times and almost ongoing continuously on source/destination clusters.
      Are there any other stats/memory that I should check for this?

      • Every time compaction kicks in, stats on destination become choppy and replication becomes slower, due to resource contention.
      • XDCR by iteself has very high CPU utilization ~90% on source.

      Screenshots of cluster overview on source and destination.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Alk,

        was something changes recently in compaction manager ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - Alk, was something changes recently in compaction manager ?
        Hide
        ketaki Ketaki Gangal added a comment -

        How much data size/memory savings do we get with compaction? ie it should reduce significantly with every compaction, right?

        Show
        ketaki Ketaki Gangal added a comment - How much data size/memory savings do we get with compaction? ie it should reduce significantly with every compaction, right?
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        depends on the type of workload you are running.
        we dont save any space in memory , compaction compacts the data which is stored on disk
        for instance if you load 1M items and delete half of them without compaction we will have 1.5 M items on the disk because we only do append only writes to disk and when compaction kicks in it deletes those items which are not valid anymore.

        Show
        farshid Farshid Ghods (Inactive) added a comment - depends on the type of workload you are running. we dont save any space in memory , compaction compacts the data which is stored on disk for instance if you load 1M items and delete half of them without compaction we will have 1.5 M items on the disk because we only do append only writes to disk and when compaction kicks in it deletes those items which are not valid anymore.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        There's not enough data.

        For example compaction may be slower then usual, then given we compact vbucket after vbucket, some vbuckets at the end of compaction would already have lots of 'garbage' thus frequent compactions. For example.

        I suggest observing all stats we have. Disk sizes & fragmentation stats particularly.

        I can also try reproducing it myself.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - There's not enough data. For example compaction may be slower then usual, then given we compact vbucket after vbucket, some vbuckets at the end of compaction would already have lots of 'garbage' thus frequent compactions. For example. I suggest observing all stats we have. Disk sizes & fragmentation stats particularly. I can also try reproducing it myself.
        Hide
        chiyoung Chiyoung Seo added a comment -

        This issue is not related to ep-engine as the compaction is totally controlled by the cluster manager and couchstore layer

        Show
        chiyoung Chiyoung Seo added a comment - This issue is not related to ep-engine as the compaction is totally controlled by the cluster manager and couchstore layer
        Hide
        xiaoqin Xiaoqin Ma (Inactive) added a comment -

        Are there are lot of updates/create/delete going on? After each compaction finishes, what is the fraction ratio? If there are not much updates/create/delete items while compaction is running, normally when compaction is done, the fraction rate should become much smaller.
        But with XDCR setup, if the fractions are caused by deletion mainly and those deletions haven't been propagated to destination yet, compaction will not really compact the data.

        Show
        xiaoqin Xiaoqin Ma (Inactive) added a comment - Are there are lot of updates/create/delete going on? After each compaction finishes, what is the fraction ratio? If there are not much updates/create/delete items while compaction is running, normally when compaction is done, the fraction rate should become much smaller. But with XDCR setup, if the fractions are caused by deletion mainly and those deletions haven't been propagated to destination yet, compaction will not really compact the data.

          People

          • Assignee:
            ketaki Ketaki Gangal
            Reporter:
            ketaki Ketaki Gangal
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes