Details
-
Improvement
-
Resolution: Fixed
-
Major
-
None
-
None
-
0
Description
In a customer issue we found cbbackupmgr had hanged and the stacktrace pointed to DispatchDirectToAddress:
goroutine 1 [runnable]:
|
github.com/couchbase/gocbcore/v10.(*memdPipeline).sendRequest(0x1e71b00?, 0xc0001e8c00?, 0xc0001e8c00?)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.0/memdpipeline.go:89 +0x45
|
github.com/couchbase/gocbcore/v10.(*memdPipeline).SendRequest(...)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.0/memdpipeline.go:105
|
github.com/couchbase/gocbcore/v10.(*kvMux).DispatchDirectToAddress(0xc23ea94930, 0xc0000430e0, 0xc1fcb81b00)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.0/kvmux.go:403 +0xcb
|
github.com/couchbase/gocbcore/v10.(*statsComponent).Stats(0xc23e1f9200, {{0x1a68a0c, 0xf}, {0x0, 0x0}, {0x1e74980, 0x2a719a8}, {0x0, 0x0, 0x0}, ...}, ...)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.0/statscomponent.go:156 +0x8e5
|
github.com/couchbase/gocbcore/v10.(*Agent).Stats(0x33c?, {{0x1a68a0c, 0xf}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
|
/home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.0/agent_ops.go:146 +0x58
|
github.com/couchbase/backup/couchbase.getBucketSeqNos({0x1e8a498, 0xc23e9bcd80}, 0x400, 0x0, 0x380?)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/backup/couchbase/sequence_numbers.go:143 +0x194
|
Whilst looking into this we found that it is possible for the loop inside that method to loop infinitely:
for {
|
err := pipeline.SendRequest(req)
|
if err == errPipelineClosed {
|
continue
|
}
|
// <snip>
|
}
|
if the pipeline is closed. Closing a pipeline is irreversible (i.e. you need to create a new one). That means once this if condition is met it will always be executed because whether the pipeline is closed is the first thing we check when trying to send a request (1, 2).
Attachments
Issue Links
- relates to
-
MB-60321 [CBM] Pick up gocbcore fix to prevent hang when gathering bucket high seqnos
-
- Closed
-
-
AV-74031 Loading...
-