Applying the same logic to the timeframe in which rebalance has failed:
Timekeeper sent a initiateRecovery for bucket STOCK
2019-11-29T06:06:22.081-08:00 [Info] Timekeeper::initiateRecovery StreamId MAINT_STREAM Bucket STOCK SessionId 1 RestartTs bucket: STOCK, vbuckets:1024 Crc64: 14463349105642428561 snapType INMEM_SNAP -
|
However, indexer was stuck waiting to process rollback for bucket WAREHOUSE (Index 13355470894422758037 belongs to bucket WAREHOUSE)
2019-11-29T06:06:22.123-08:00 [Info] StorageMgr::handleRollback Rollback Index: 13355470894422758037 PartitionId: 0 SliceId: 0 To Snapshot SnapshotInfo: count:3 committed:false
|
Indexer got prepare topology change at 2019-11-29T06:06:27.320-08:00
2019-11-29T06:06:27.320-08:00 [Info] ServiceMgr::PrepareTopologyChange
|
As a part of prepareTopologyChange, rebalance service manager send a message to indexer to check DDL running. However, it will be processed after the storage finishes rollback for bucket WAREHOUSE and indexer processes initateRecovery for bucket STOCK.In the mean time, there have been multiple initiateRecovery requests from timekeeper
2019-11-29T06:06:32.427-08:00 [Info] Timekeeper::initiateRecovery StreamId MAINT_STREAM Bucket NEW_ORDER SessionId 3 RestartTs bucket: NEW_ORDER, vbuckets: 1024 Crc64: 15483437431460244926 snapType INMEM_SNAP -
|
2019-11-29T06:06:32.429-08:00 [Info] Timekeeper::initiateRecovery StreamId MAINT_STREAM Bucket ITEM SessionId 3 RestartTs bucket: ITEM, vbuckets: 1024 Crc64: 11607813187078538988 snapType INMEM_SNAP -
|
2019-11-29T06:06:32.430-08:00 [Info] Timekeeper::initiateRecovery StreamId MAINT_STREAM Bucket HISTORY SessionId 1 RestartTs bucket: HISTORY, vbuckets: 1024 Crc64: 17135845864963193340 snapType INMEM_SNAP -
|
2019-11-29T06:06:32.432-08:00 [Info] Timekeeper::initiateRecovery StreamId MAINT_STREAM Bucket ORDERS SessionId 1 RestartTs bucket: ORDERS, vbuckets: 1024 Crc64: 7597314664672643693 snapType INMEM_SNAP -
|
Indexer could process the initateRecovery request for bucket STOCK at 2019-11-29T06:06:33.406-08:00
2019-11-29T06:06:33.406-08:00 [Info] Indexer::handleInitRecovery StreamId MAINT_STREAM Bucket STOCK STREAM_RECOVERY
|
Immediately, after this, the checkDDL running was processed by indexer:
2019-11-29T06:06:33.427-08:00 [Info] ServiceMgr::prepareRebalance Found DDL Running []
|
2019-11-29T06:06:33.427-08:00 [Info] ServiceMgr::prepareRebalance Init Prepare Phase
|
As a part of prepare topology change, rebalance manager would send another request to cluster manager via indexer. This request would be processed after all initiateRecovery requests for different buckets are processed. This happened at 2019-11-29T06:07:20.822-08:00. By this time rebalance timed out
2019-11-29T06:07:20.822-08:00 [Info] ClustMgr:handleSetLocalValue Key RebalanceRunning Value
|
2019-11-29T06:07:20.823-08:00 [Info] ClustMgr:handleRebalanceRunning &\{61 [] {false false false false false false false} 0 false
|
No changes related to rebalance in GSI between the mentioned builds.
CHANGELOG for indexing
Merge remote-tracking branch 'couchbase/unstable' into HEAD
http://ci2i-unstable.northscale.in/gsi-26.11.2019-16.01.pass.html
Change-Id: I3d26db6e7f1ae8f5d3b8e5887eddc8945eb1f38a
MB-36964: Fix recoverableCreateIndex in case of planner errorIf planner fails to generate a solution that satisfies the
connstraints, a round robin approach is used to allow index
creation. But the layout generated by the round robin approach
is not used during commit phase.
The fix ensures that the layout generated by the round robin
approach is used during commit phase.
Change-Id: Ie108946cb75c0d81e3c0d92a5742c7e062866531