Details
-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Security Level: Public
Description
Defining a replicator on a database then reloading that database will cause an unrecoverable panic. In the CBSE, the database was being reloaded due to network issues.
The panic is to do with trying to register duplicate Prometheus stats.
Repro steps on 1 SG instance
1. Create db1 with bucket1
2. Create db2 with bucket2
3. Create a replicator on db2 with db1 as remote. Can be pull or push and can be continuous or not.
4. Reload database for db2. I did this by modifying db2 config through PUT /db2/_config by adding "import_docs": true to the config.
5. Wait for panic
Repro URL and payloads at bottom
3.1.0 panic message
panic: duplicate metrics collector registration attempted
|
|
|
goroutine 11404 [running]:
|
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0bcba6a00?, {0xc0bcba8e40?, 0x1, 0x4f084d8?})
|
/Users/isaaclambat/go/pkg/mod/github.com/prometheus/client_golang@v1.11.1/prometheus/registry.go:403 +0x7f
|
github.com/prometheus/client_golang/prometheus.MustRegister(...)
|
/Users/isaaclambat/go/pkg/mod/github.com/prometheus/client_golang@v1.11.1/prometheus/registry.go:178
|
github.com/couchbase/sync_gateway/base.NewIntStat({0x4f084d8?, 0xc0e1caddb5?}, {0x4f26fd7?, 0x40125e5?}, {0xc00099dc40?, 0x12?, 0xc0179e6360?}, {0xc0bcba6a00, 0x2, 0x2}, ...)
|
/Users/isaaclambat/dev/sync-gateway/master/sync_gateway/base/stats.go:617 +0x133
|
github.com/couchbase/sync_gateway/base.(*DbStats).DBReplicatorStats(0xc0b5554fc0, {0xc0e1caf1b5, 0x5})
|
/Users/isaaclambat/dev/sync-gateway/master/sync_gateway/base/stats.go:1097 +0x1ba
|
github.com/couchbase/sync_gateway/db.(*sgReplicateManager).InitializeReplication(0xc0b7cb0120, 0xc0d4466b40)
|
/Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/sg_replicate_cfg.go:661 +0x25f
|
github.com/couchbase/sync_gateway/db.(*sgReplicateManager).StartReplications(0xc0b7cb0120)
|
/Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/sg_replicate_cfg.go:508 +0x22a
|
github.com/couchbase/sync_gateway/db.(*DatabaseContext).StartReplications.func1()
|
/Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/sg_replicate_cfg.go:445 +0x285
|
created by github.com/couchbase/sync_gateway/db.(*DatabaseContext).StartReplications
|
/Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/sg_replicate_cfg.go:429 +0x10a
|
Repro steps URLs and payloads
PUT 127.0.0.1:4985/db1/
{
|
"num_index_replicas": 0,
|
"bucket": "bucket1"
|
}
|
PUT 127.0.0.1:4985/db2/
{
|
"num_index_replicas": 0,
|
"bucket": "bucket2"
|
}
|
PUT 127.0.0.1:4985/db2/_replication/repl1
{
|
"id":"repl1",
|
"remote":"http://127.0.0.1:4985/db1",
|
"direction":"pull",
|
"continuous": true
|
}
|
PUT 127.0.0.1:4985/db2/_config
{
|
"bucket": "bucket2",
|
"name": "db2",
|
"num_index_replicas": 0,
|
"import_docs": true
|
}
|