Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Unknown
Description
Build : 7.0.0-5161
Test : -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
Scale : 3
Iteration : 1st
Rebalance to add back a failed over and recovered FTS node 172.23.104.157 is stuck for 9+ hrs.
Test console:
[2021-05-14T15:19:20-07:00, sequoiatools/couchbase-cli:7.0:911c05] failover -c 172.23.108.103:8091 --server-failover 172.23.104.157:8091 -u Administrator -p password --hard
|
[2021-05-14T15:19:28-07:00, sequoiatools/couchbase-cli:7.0:e3d528] recovery -c 172.23.108.103:8091 --server-recovery 172.23.104.157:8091 --recovery-type full -u Administrator -p password
|
[2021-05-14T15:19:33-07:00, sequoiatools/couchbase-cli:7.0:6a02c7] rebalance -c 172.23.108.103:8091 -u Administrator -p password
|
On 172.23.104.157, some warnings like the following can be seen after which rebalance appears to be stuck -
2021-05-14T15:23:04.316-07:00 [WARN] janitor: JanitorOnce, err: janitor: JanitorOnce errors: 1, []string{"#0: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not start feed: bucket_bucket7_idx_cehjd-country-price_7266521f5adc7ca9_b6d0c5f9, server: http://127.0.0.1:8091, err: Start, name: bucket_bucket7_idx_cehjd-country-price_7266521f5adc7ca9_b6d0c5f9, vbid: 628, err: bleve: BleveDest already closed"} -- cbgt.(*Manager).JanitorLoop() at manager_janitor.go:97
|
2021-05-14T15:23:04.316-07:00 [WARN] feed_dcp_gocbcore: [bucket_bucket7_idx_cehjd-country-price_7266521f5adc7ca9_b6d0c5f9] Rollback to seqno: 0, vbuuid: 0 for vb: 530, failed with err: bleve: BleveDest already closed -- cbgt.(*GocbcoreDCPFeed).rollback() at feed_dcp_gocbcore.go:1533
|
2021-05-14T15:23:04.316-07:00 [WARN] feed_dcp_gocbcore: [bucket_bucket7_idx_cehjd-country-price_7266521f5adc7ca9_b6d0c5f9] Rollback to seqno: 0, vbuuid: 0 for vb: 541, failed with err: bleve: BleveDest already closed -- cbgt.(*GocbcoreDCPFeed).rollback() at feed_dcp_gocbcore.go:1533
|
2021-05-14T15:23:04.316-07:00 [WARN] feed_dcp_gocbcore: [bucket_bucket7_idx_cehjd-country-price_7266521f5adc7ca9_b6d0c5f9] Rollback to seqno: 0, vbuuid: 0 for vb: 555, failed with err: bleve: BleveDest already closed -- cbgt.(*GocbcoreDCPFeed).rollback() at feed_dcp_gocbcore.go:1533
|
2021-05-14T15:23:04.316-07:00 [WARN] feed_dcp_gocbcore: [bucket_bucket7_idx_cehjd-country-price_7266521f5adc7ca9_b6d0c5f9] Rollback to seqno: 0, vbuuid: 0 for vb: 546, failed with err: bleve: BleveDest already closed -- cbgt.(*GocbcoreDCPFeed).rollback() at feed_dcp_gocbcore.go:1533
|
2021-05-14T15:23:04.316-07:00 [WARN] feed_dcp_gocbcore: [bucket_bucket7_idx_cehjd-country-price_7266521f5adc7ca9_b6d0c5f9] Rollback to seqno: 0, vbuuid: 0 for vb: 543, failed with err: bleve: BleveDest already closed -- cbgt.(*GocbcoreDCPFeed).rollback() at feed_dcp_gocbcore.go:1533
|
FTS nodes : 172.23.104.155, 172.23.104.157, 172.23.96.148
Attachments
| For Gerrit Dashboard: MB-46312 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 153791,8 | MB-46312 - Adding config to unblock a stuck rebalance | master | cbgt | Status: MERGED | +2 | +1 |
| 153849,2 | MB-46312: Record seqno received within a seqno_advanced message | master | cbft | Status: MERGED | +2 | +1 |
| 153971,6 | MB-46312: Extend statsWriter to obtain indexDef, sourceSeqs | master | cbgt | Status: MERGED | +2 | +1 |
| 153976,6 | MB-46312: Record "sourceSeq" alongside "seqReceived" within pindex stats | master | cbft | Status: MERGED | +2 | +1 |
| 154006,5 | MB-46312: Rebalance progress is stuck indefinitely | master | cbgt | Status: MERGED | +2 | +1 |