[System Test] Index rebalance failure due to error - "index build is in progress for indexes: [bucket4:idx1_FMbG3g..."

Rebalance exited with reason {service_rebalance_failed,index,

{worker_died,

{'EXIT',<0.25943.1731>,

{rebalance_failed,

{service_error,

<<"indexer rebalance failure - index build is in progress for indexes: [bucket4:idx1_FMbG3g bucket4:idx1_3uZA bucket4:idx2_9Tw9Yht bucket4:idx3_9mcwgX bucket4:idx1_aFWzKbDa bucket4:idx2_ftIQ0Tnu bucket4:idx3_NcVjBL3s bucket4:idx3_iDBWps bucket4:idx2_qXidYRP bucket4:idx3_fGeZ8Rzz bucket6:idx1_l4TJM bucket4:idx2_WOitrF bucket6:idx1_l4TJM bucket6:idx3_Rpd71n bucket4:idx2_yO84 bucket4:idx2_WOitrF bucket4:idx3_8JDSl0rZ bucket4:idx3_9mcwgX bucket4:idx3_9mcwgX bucket4:idx1_5maxvD5].">>}}}}}.

Rebalance Operation Id = 7beaa99284d647ea0fcd458ddc1471b5

[root@xcp-s205-mgt logs]# zgrep 2021-03-14T16:22:58 indexer.log* | grep "\[Error"

indexer.log.2.gz:2021-03-14T16:22:58.103-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.103-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [14163549755496642964 12922949540434451582 673475531003878329 4563390483730277566 187746286261005158]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.103-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[187746286261005158:Indexer Cannot Process Build Index - Rebalance In Progress 673475531003878329:Indexer Cannot Process Build Index - Rebalance In Progress 4563390483730277566:Indexer Cannot Process Build Index - Rebalance In Progress 12922949540434451582:Indexer Cannot Process Build Index - Rebalance In Progress 14163549755496642964:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.225-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.225-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [16949110365549165662 12545405990327357155 6607573522748631142]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.225-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[6607573522748631142:Indexer Cannot Process Build Index - Rebalance In Progress 12545405990327357155:Indexer Cannot Process Build Index - Rebalance In Progress 16949110365549165662:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.349-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.349-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [9758557981145163521 2950145673377536614]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.349-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[2950145673377536614:Indexer Cannot Process Build Index - Rebalance In Progress 9758557981145163521:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.359-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.359-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [3033602912840348077 10872749546174730804]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.359-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[3033602912840348077:Indexer Cannot Process Build Index - Rebalance In Progress 10872749546174730804:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.373-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.373-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [16949110365549165662 12545405990327357155 6607573522748631142]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.373-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[6607573522748631142:Indexer Cannot Process Build Index - Rebalance In Progress 12545405990327357155:Indexer Cannot Process Build Index - Rebalance In Progress 16949110365549165662:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.385-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.385-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [17440055532778370888 6708111709481081276]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.385-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[6708111709481081276:Indexer Cannot Process Build Index - Rebalance In Progress 17440055532778370888:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.397-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.397-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [4616930540165911076 11271808093427277782]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.397-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[4616930540165911076:Indexer Cannot Process Build Index - Rebalance In Progress 11271808093427277782:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.408-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.408-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [4241597596603636577 4602120942921108340]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.408-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[4241597596603636577:Indexer Cannot Process Build Index - Rebalance In Progress 4602120942921108340:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.440-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.440-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [2595614399445880639 12694065441132671700 4114260517856832096]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.440-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[2595614399445880639:Indexer Cannot Process Build Index - Rebalance In Progress 4114260517856832096:Indexer Cannot Process Build Index - Rebalance In Progress 12694065441132671700:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.463-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.463-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [13981434716248606315 2699404461264428244]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.463-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[2699404461264428244:Indexer Cannot Process Build Index - Rebalance In Progress 13981434716248606315:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.475-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.475-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [2608084710622109065 13483372737282728073]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.475-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[2608084710622109065:Indexer Cannot Process Build Index - Rebalance In Progress 13483372737282728073:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.501-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.501-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [1769313993583335507 14713797754005520097 17521682677576093054]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.501-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[1769313993583335507:Indexer Cannot Process Build Index - Rebalance In Progress 14713797754005520097:Indexer Cannot Process Build Index - Rebalance In Progress 17521682677576093054:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.515-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.515-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [9309540688918072280 4269995515739264907]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.515-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[4269995515739264907:Indexer Cannot Process Build Index - Rebalance In Progress 9309540688918072280:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.538-07:00 [Error] Indexer::handleBuildIndex Indexer Cannot Process Build Index - Rebalance In Progress

indexer.log.2.gz:2021-03-14T16:22:58.538-07:00 [Error] clustMgrAgent::OnIndexBuild Error for Build Index [15513659224000895412 13469626796295834242]. Error Indexer Cannot Process Build Index - Rebalance In Progress.

indexer.log.2.gz:2021-03-14T16:22:58.538-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[13469626796295834242:Indexer Cannot Process Build Index - Rebalance In Progress 15513659224000895412:Indexer Cannot Process Build Index - Rebalance In Progress]

indexer.log.2.gz:2021-03-14T16:22:58.921-07:00 [Error] Indexer::checkParallelCollectionBuilds [11240448337146520357 3367730938630713917], bucket7:scope_0:coll_2. Current collection build map: map[bucket4:scope_2:coll_6:true bucket4:scope_3:coll_1:true bucket5:scope_1:coll_5:true bucket5:scope_2:coll_0:true bucket5:scope_3:coll_1:true bucket6:scope_0:coll_6:true bucket6:scope_2:coll_3:true bucket6:scope_2:coll_4:true bucket6:scope_2:coll_6:true bucket6:scope_2:coll_7:true]

indexer.log.2.gz:2021-03-14T16:22:58.921-07:00 [Error] Indexer::handleBuildIndex Build is already in progress for 10 collections. KeyspaceID: [11240448337146520357 3367730938630713917]. Instances in error: bucket7:scope_0:coll_2

indexer.log.2.gz:2021-03-14T16:22:58.921-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[3367730938630713917:Build Already In Progress for 10 collections. 11240448337146520357:Build Already In Progress for 10 collections.]

indexer.log.2.gz:2021-03-14T16:22:58.942-07:00 [Error] Indexer::checkParallelCollectionBuilds [14163549755496642964 12922949540434451582 673475531003878329 4563390483730277566 187746286261005158], bucket7:scope_3:coll_7. Current collection build map: map[bucket4:scope_2:coll_6:true bucket4:scope_3:coll_1:true bucket5:scope_1:coll_5:true bucket5:scope_2:coll_0:true bucket5:scope_3:coll_1:true bucket6:scope_0:coll_6:true bucket6:scope_2:coll_3:true bucket6:scope_2:coll_4:true bucket6:scope_2:coll_6:true bucket6:scope_2:coll_7:true]

indexer.log.2.gz:2021-03-14T16:22:58.942-07:00 [Error] Indexer::handleBuildIndex Build is already in progress for 10 collections. KeyspaceID: [14163549755496642964 12922949540434451582 673475531003878329 4563390483730277566 187746286261005158]. Instances in error: bucket7:scope_3:coll_7

indexer.log.2.gz:2021-03-14T16:22:58.942-07:00 [Error] LifecycleMgr.handleBuildIndexes() : buildIndex fails. Reason = map[187746286261005158:Build Already In Progress for 10 collections. 673475531003878329:Build Already In Progress for 10 collections. 4563390483730277566:Build Already In Progress for 10 collections. 12922949540434451582:Build Already In Progress for 10 collections. 14163549755496642964:Build Already In Progress for 10 collections.]