Description
A bucket topology interval ticker is shared among many subscribers. Right now, when a subscriber subscribes, the ticker is reset to the lowest interval.
This could lead to reseting the timer ticker too often, and resulting in launching too many non-blocking go-routines since each time a Reset() call occurs, the timer unblocks and does not wait for the launched go-routine to finish.
A example of such symptom can be found in the goxdcr_mprof.log where it shows a large heap and the top of it is a large number of go-routines that do the same thing, which shouldn't have been possible unless ticker.Reset() is called.
Note the below stack trace shows too many goroutines with only 4 watchers (4 source buckets), each watcher doing duplicate go-routine work.
heap profile: 10620: 2291184720 [742729: 17437853416] @ heap/1048576
|
267: 1264238592 [451: 2135474176] @ 0x411a99 0x4136e9 0x416c4a 0x74c1c8 0xaeb37e 0xac3824 0xaeec74 0x471841
|
# 0x74c1c7 github.com/couchbase/gomemcached/client.(*Client).StatsMap+0x227 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/client/mc.go:1392
|
# 0xaeb37d github.com/couchbase/goxdcr/service_impl.(*BucketTopologyService).getDcpStatsUpdater.func1+0x33d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:195
|
# 0xac3823 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).updateOnce+0x103 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1101
|
# 0xaeec73 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func2+0x253 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:944
|
|
333: 788373504 [1318: 3120349184] @ 0x411a99 0x4136e9 0x416c4a 0x74c1c8 0xaeb37e 0xac3824 0xaeec74 0x471841
|
# 0x74c1c7 github.com/couchbase/gomemcached/client.(*Client).StatsMap+0x227 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/client/mc.go:1392
|
# 0xaeb37d github.com/couchbase/goxdcr/service_impl.(*BucketTopologyService).getDcpStatsUpdater.func1+0x33d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:195
|
# 0xac3823 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).updateOnce+0x103 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1101
|
# 0xaeec73 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func2+0x253 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:944
|
|
465: 140943360 [1367: 414343168] @ 0x411a99 0x4136e9 0x416c4a 0x74c1c8 0xaeb37e 0xac3824 0xaeec74 0x471841
|
# 0x74c1c7 github.com/couchbase/gomemcached/client.(*Client).StatsMap+0x227 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/client/mc.go:1392
|
# 0xaeb37d github.com/couchbase/goxdcr/service_impl.(*BucketTopologyService).getDcpStatsUpdater.func1+0x33d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:195
|
# 0xac3823 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).updateOnce+0x103 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1101
|
# 0xaeec73 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func2+0x253 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:944
|
|
68: 40665088 [1439: 860545024] @ 0x411a99 0x4136e9 0x416c4a 0x74c1c8 0xaeb37e 0xac3824 0xaeec74 0x471841
|
# 0x74c1c7 github.com/couchbase/gomemcached/client.(*Client).StatsMap+0x227 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/client/mc.go:1392
|
# 0xaeb37d github.com/couchbase/goxdcr/service_impl.(*BucketTopologyService).getDcpStatsUpdater.func1+0x33d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:195
|
# 0xac3823 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).updateOnce+0x103 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1101
|
# 0xaeec73 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func2+0x253 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:944
|
|
9: 10690560 [1286: 1527562240] @ 0x411a99 0x4136e9 0x416c4a 0x74c1c8 0xaeb37e 0xac3824 0xaeec74 0x471841
|
# 0x74c1c7 github.com/couchbase/gomemcached/client.(*Client).StatsMap+0x227 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/client/mc.go:1392
|
# 0xaeb37d github.com/couchbase/goxdcr/service_impl.(*BucketTopologyService).getDcpStatsUpdater.func1+0x33d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:195
|
# 0xac3823 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).updateOnce+0x103 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:1101
|
# 0xaeec73 github.com/couchbase/goxdcr/service_impl.(*BucketTopologySvcWatcher).run.func2+0x253 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/bucketTopologyService.go:944
|
|
155: 5079040 [25935: 849838080] @ 0x681110 0x682085 0x7551fe 0x471841
|
# 0x68110f github.com/couchbase/gomemcached.(*MCRequest).populateRegularBody+0x6f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/mc_req.go:488
|
# 0x682084 github.com/couchbase/gomemcached.(*MCRequest).Receive+0x204 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/mc_req.go:645
|
# 0x7551fd github.com/couchbase/gomemcached/client.(*UprFeed).runFeed+0x19d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gomemcached/client/upr_feed.go:878
|
|
Attachments
Issue Links
- is triggering
-
MB-48323 [System Test] error retrieving default pool info from target - warnings observed when refreshing remote cluster reference
- Closed