Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57463

[BP 7.1.5] - XDCR - statsMgr stopping could hang due to watching for notification

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.1.5
    • 7.1.4, 7.1.0, 7.1.1, 7.1.2, 7.2.0, 7.1.3, 7.2.1
    • XDCR
    • Untriaged
    • 0
    • Unknown

    Description

      Issue Resolution
      StatsMgr stopping could hang due to watching for notifications resulting in stranded go-routines. Go-routines are now stopped correctly.

       

      I’m able to reproduce it by adding a forced sleep:

      case oneRequestor := <-stats_mgr.latestNotificationReqCh:
      	// Because the channel can length can vary, just do a green clone to be safe
      	notificationForRequestor := notification.Clone(1).(service_def.SourceNotification)
      	stats_mgr.logger.Infof("NEIL DEBUG sleeping 30 seconds\n")
      	time.Sleep(30 * time.Second)
      	oneRequestor.sendBack <- notificationForRequestor
      

      And during this 30 seconds once the message shows up, manually pause the pipeline. Then, I see the same stack traces of which I was suspicious:

      1 @ 0x10003b7f6 0x10004c9af 0x10004c986 0x10006ae67 0x10008c7cb 0x100644e45 0x10069542e 0x10006f7e1
      #       0x10006ae66     sync.runtime_Semacquire+0x26                                                    /Users/neil.huang/.cbdepscache/exploded/x86_64/go-1.20/go/src/runtime/sema.go:62
      #       0x10008c7ca     sync.(*WaitGroup).Wait+0x4a                                                     /Users/neil.huang/.cbdepscache/exploded/x86_64/go-1.20/go/src/sync/waitgroup.go:116
      #       0x100644e44     github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).Stop+0xc4         /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:985
      #       0x10069542d     github.com/couchbase/goxdcr/pipeline_ctx.(*PipelineRuntimeCtx).stopService+0xed /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_ctx/pipeline_runtimeCtx.go:201
       
      1 @ 0x10003b7f6 0x10004c9af 0x10004c986 0x10006ae67 0x10008c7cb 0x100694c4f 0x1003eba82 0x10006f7e1
      #       0x10006ae66     sync.runtime_Semacquire+0x26                                                    /Users/neil.huang/.cbdepscache/exploded/x86_64/go-1.20/go/src/runtime/sema.go:62
      #       0x10008c7ca     sync.(*WaitGroup).Wait+0x4a                                                     /Users/neil.huang/.cbdepscache/exploded/x86_64/go-1.20/go/src/sync/waitgroup.go:116
      #       0x100694c4e     github.com/couchbase/goxdcr/pipeline_ctx.(*PipelineRuntimeCtx).Stop.func2+0x38e /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_ctx/pipeline_runtimeCtx.go:170
      #       0x1003eba81     github.com/couchbase/goxdcr/base.ExecWithTimeout.func1+0x21                     /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/base/simple_utils.go:48
       
       
      1 @ 0x10003b7f6 0x100006a2e 0x1000065dd 0x100646165 0x10006f7e1
      #       0x100646164     github.com/couchbase/goxdcr/pipeline_svc.(*StatisticsManager).watchNotificationCh+0x224 /Users/neil.huang/source/couchbase/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/statistics_manager.go:1131
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              neil.huang Neil Huang
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty