Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25982

Compaction of large vbuckets causes a significant increase in memory usage

    XMLWordPrintable

    Details

      Description

      If vbuckets are large (e.g., 100M or 200M keys per vbucket), then compaction of individual vbuckets triggers a noticeable spike in memory usage.

      For instance, this relatively small vbucket temporary increased memory usage by ~15GB:

      DB Info (/data/bucket-1/0.couch.2) - header at 33279463424
         file format version: 12
         update_seq: 100000000
         purge_seq: 0
         crc: CRC-32C
         doc count: 100000000
         deleted doc count: 0
         data size: 30.993 GB
         B-tree size: 7.045 GB
         total disk size: 30.994 GB
      

      Larger vbuckets sent my 64GB servers to swap.

      The amount of RAM allocated during the compaction process depends on the size of the index. I am pretty sure we can come up with a good formula for that.

      Steps:

      • Create a cluster with 4 nodes, 1 bucket, and 4 vbuckets (no replicas)
      • Disable auto-compaction
      • Load 800M items (~512B)
      • Trigger compaction

      Some graphs: http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hera_510-1137_compact_5197

        Attachments

          Issue Links

          For Gerrit Dashboard: MB-25982
          # Subject Branch Project Status CR V

            Activity

            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-1006.5.1-1064 contains couchstore commit 5cd4091 with commit message:
            MB-25982: Reduce memory usage during compaction

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-1006.5.1-1064 contains couchstore commit 5cd4091 with commit message: MB-25982 : Reduce memory usage during compaction
            Hide
            bo-chun.wang Bo-Chun Wang added a comment -

            I'm trying to re-run the same scenario. I followed the steps provided by Pavel to create a cluster with 4 nodes, 1 bucket, and 4 vbuckets. However, my tests kept failing during the cluster creation because some nodes are not healthy. According to the log, the configuration is invalid. Can someone take a look at the log? Thanks.

            [ns_server:info,2020-03-26T14:50:57.418-07:00,babysitter_of_ns_1@cb.local:<0.193.0>:ns_port_server:log:224]memcached<0.193.0>: 2020-03-26T14:50:57.218077-07:00 WARNING (bucket-1) Invalid configuration: Shards must be equal or less than max number of vbuckets

            Build: 6.5.1-6278

            Job: http://perf.jenkins.couchbase.com/job/hera-compaction/

            Log:

            https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.117.zip

            https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.118.zip

            https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.119.zip

            https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.120.zip

            Show
            bo-chun.wang Bo-Chun Wang added a comment - I'm trying to re-run the same scenario. I followed the steps provided by Pavel to create a cluster with 4 nodes, 1 bucket, and 4 vbuckets. However, my tests kept failing during the cluster creation because some nodes are not healthy. According to the log, the configuration is invalid. Can someone take a look at the log? Thanks. [ns_server:info,2020-03-26T14:50:57.418-07:00,babysitter_of_ns_1@cb.local:<0.193.0>:ns_port_server:log:224] memcached<0.193.0>: 2020-03-26T14:50:57.218077-07:00 WARNING (bucket-1) Invalid configuration: Shards must be equal or less than max number of vbuckets Build: 6.5.1-6278 Job:  http://perf.jenkins.couchbase.com/job/hera-compaction/ Log: https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.117.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.118.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.119.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2020-03-26T215934-ns_1%40172.23.96.120.zip
            Hide
            ben.huddleston Ben Huddleston added a comment -

            Hey Bo-Chun Wang. If you want to set 4 vBucket then you need 4 or fewer shards at the moment (max_num_shards). MB-38008 will change this at some point in the future.

            Show
            ben.huddleston Ben Huddleston added a comment - Hey Bo-Chun Wang . If you want to set 4 vBucket then you need 4 or fewer shards at the moment (max_num_shards). MB-38008 will change this at some point in the future.
            Show
            bo-chun.wang Bo-Chun Wang added a comment - I finished two runs. Compared to the baseline run, there is no spike in memory usage in the compare run. The fix is verified. I close this ticket.  Baseline: 6.5.0-4960 http://perf.jenkins.couchbase.com/job/hera-compaction/3/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hera_650-4960_compact_ac0d#c85f8d6214cea4764963b03428157303 Compare: 6.5.1-6278 http://perf.jenkins.couchbase.com/job/hera-compaction/2/ http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=hera_651-6278_compact_a511#c85f8d6214cea4764963b03428157303
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-1006.5.1-1125 contains couchstore commit 5cd4091 with commit message:
            MB-25982: Reduce memory usage during compaction

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-1006.5.1-1125 contains couchstore commit 5cd4091 with commit message: MB-25982 : Reduce memory usage during compaction

              People

              Assignee:
              bo-chun.wang Bo-Chun Wang
              Reporter:
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  PagerDuty