Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58671

XDCR Checkpoint manager and pipeline stop in goroutine

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.6.0
    • Morpheus, 7.2.1, 7.1.5, 7.1.6
    • XDCR
    • Untriaged
    • 0
    • Unknown

    Description

      Observing checkpoint manager stop and pipeline stop log in goroutines in recent run, leading to small memory leak:

      2 @ 0x43f4b6 0x450b6f 0x450b46 0x46db27 0x48ea8b 0x9e5965 0xa6234e 0x471e01
      #    0x46db26    sync.runtime_Semacquire+0x26                            /home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/runtime/sema.go:62
      #    0x48ea8a    sync.(*WaitGroup).Wait+0x4a                            /home/couchbase/.cbdepscache/exploded/x86_64/go-1.20.6/go/src/sync/waitgroup.go:116
      #    0x9e5964    github.com/couchbase/goxdcr/pipeline_svc.(*CheckpointManager).Stop+0x44        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/checkpoint_manager.go:843
      #    0xa6234d    github.com/couchbase/goxdcr/pipeline_ctx.(*PipelineRuntimeCtx).stopService+0xed    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_ctx/pipeline_runtimeCtx.go:201 

      goroutine file: goroutine_172.23.104.19.txt

      Steps to reproduce:
      1: Setup two 2-node clusters with 10 buckets each.
      2. Start loading documents in all buckets.
      3. Setup 100 replications between the clusters.
      4. Introduce network connectivity issues between the clusters (block incoming traffic from target cluster).
      5. Slight memory leak observed.

      Attaching logs below:
      Source cluster logs:
      https://cb-engineering.s3.amazonaws.com/MB-58539_src/collectinfo-2023-09-14T134940-ns_1%40172.23.104.19.zip
      https://cb-engineering.s3.amazonaws.com/MB-58539_src/collectinfo-2023-09-14T134940-ns_1%40172.23.104.21.zip
      https://cb-engineering.s3.amazonaws.com/MB-58539_src/collectinfo-2023-09-14T134940-ns_1%40172.23.104.23.zip

      Target cluster logs:
      https://cb-engineering.s3.amazonaws.com/MB-58539_dest/collectinfo-2023-09-14T135006-ns_1%40172.23.104.18.zip
      https://cb-engineering.s3.amazonaws.com/MB-58539_dest/collectinfo-2023-09-14T135006-ns_1%40172.23.104.17.zip
      https://cb-engineering.s3.amazonaws.com/MB-58539_dest/collectinfo-2023-09-14T135006-ns_1%40172.23.104.16.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ayush.nayyar Ayush Nayyar
              ayush.nayyar Ayush Nayyar
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty