[System Test] Panics observed in indexer post stopping rebalance operation

2024-01-23T22:58:34.862-08:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [    InstId: 14708763738525686602    Defn: DefnId: 17262890022861652964 Name: idx8_oW9L3f Using: plasma Bucket: bucket1 Scope/Id: scope_0/8 Collection/Id: coll_12/14 IsPrimary: false NumReplica: 2 InstVersion: 2         SecExprs: <ud>([(distinct (array flatten_keys((`r`.`author`), ((`r`.`ratings`).`Cleanliness`)) for `r` in `reviews` when (((`r`.`ratings`).`Cleanliness`) < 4) end)) `country` `email` `free_parking`])</ud>         Desc: [false false false false false]        IndexMissingLeadingKey: false        IsPartnKeyDocId: true        PartitionScheme: KEY         HashScheme: CRC32 PartitionKeys: [(meta().`id`)] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false         AlternateShardIds: map[4:[7633304017478618078-1-0 7633304017478618078-1-1]]     State: INDEX_STATE_CATCHUP    RState: RebalPending    Stream: INIT_STREAM    Version: 2    ReplicaId: 1    RealInstId: 12497970581884633758    PartitionContainer: &{map[4:{4 2 [:9105] []}] 4 KEY 0}] bucket:  scope:  collection:  streamId: NIL_STREAM syncUpdate: false respCh: <nil> 2024-01-23T22:58:34.865-08:00 [Info] send progress stats to clients2024-01-23T22:58:34.865-08:00 [Error] Indexer::processBuildDoneCatchup Fail to find real index instance 12497970581884633758panic: Fail to find real index instance 12497970581884633758

goroutine 1 [running]:github.com/couchbase/indexing/secondary/common.CrashOnError(...)    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:509github.com/couchbase/indexing/secondary/indexer.(*indexer).processBuildDoneCatchup(0xc0003bcc00, 0x3, {0xc00635f188, 0x17}, 0x4e91, 0xc018ef5790)    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7058 +0x1638github.com/couchbase/indexing/secondary/indexer.(*indexer).handleInitialBuildDone(0xc0003bcc00, {0x1fa0ce0?, 0xc02cca8140?})    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6932 +0x370github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0xc0003bcc00, {0x1fa0ce0, 0xc02cca8140})    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1442 +0x8bagithub.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc0003bcc00)    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1184 +0x348github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x2ebef80?)    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:607 +0x2c0cmain.main()    goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:207 +0x12322024-01-23T22:58:35.380-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=b2a29200f37518b01450080405f58afe -isEnterprise=true --deploymentModel=provisioned]2024-01-23T22:58:35.380-08:00 [Info] Indexer DeploymentModel is set to: provisioned