Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58369

Increased memory usage for cbgt 1.3.4 vs 1.3.2

    XMLWordPrintable

Details

    • Task
    • Resolution: Resolved
    • Major
    • None
    • None
    • cbgt
    • None
    • 0

    Description

      Sync Gateway is showing a significant increase in the number of gocbcore pipeline read buffers when running with cbgt 1.3.4 vs. cbgt 1.3.2. I've isolated the memory difference to the cbgt version - we see the difference in the heap profile when switching the cbgt version, and keeping everything else constant.

      The memory is associated with gocbcore initializing a memcached connection - it's the number of 20MB read buffers being initialized by gocbcore. I'm assuming this is related to how cbgt is initializing DCP agents.

      I've tested with the following configuration, with cbgt 1.3.2 and 1.3.4:

      • one node of Sync Gateway
      • one node of Couchbase Server 7.1.2
      • Sync Gateway running a cbgt gocbcore DCP feed with 16 partitions, with all partitions assigned to the single SG node

      cbgt 1.3.2:
      6 gocbcore read buffers (120 MB)

      cbgt 1.3.4:
      24 gocbcore read buffers (480 MB)

      I'll attach Sync Gateway debug logs for these two tests, which includes cbgt logging. I've also included the heap profiles showing the difference in heap size.

      The difference in memory is more significant when running with multiple Couchbase Server nodes, as the number of read buffers is multiplied by the number of server nodes (as seen in perf tests on CBG-3283).

      Do you know of any changes between these cbgt versions that could account for this difference?

      Attachments

        1. heap_132.png
          heap_132.png
          532 kB
        2. heap_132.pprof
          13 kB
        3. heap_134.png
          heap_134.png
          350 kB
        4. heap_134.pprof
          21 kB
        5. Screenshot 2023-08-24 at 4.05.44 PM.png
          Screenshot 2023-08-24 at 4.05.44 PM.png
          356 kB
        6. sg_cbgt_1.3.2.log
          150 kB
        7. sg_cbgt_1.3.4.log
          101 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              abhinav Abhi Dangeti
              adamf Adam Fraser
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty