[System Test] Indexer panic - both proxy index Instance and real index instance not found

2023-11-17T01:18:47.036-08:00 [Warn] MergePartition: Source Index Instance 3427527405380564430 is not in bucket bucket2.  Do not merge now.

2023-11-17T01:18:47.045-08:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_UPDATE_INDEX_INST elapsed 11.290608ms len(expediates) 1 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error <nil>

2023-11-17T01:18:47.045-08:00 [Info] LifecycleMgr.MergePartition() : index defnId 5371159320753348579 source 9655805858047935938 target 6054010877007640190

2023-11-17T01:18:47.045-08:00 [Info] ShardRebalancer::destTokenToMergeOrReady Response reveived for merge of index instance: 2386855181665111430, realInst: 6054010877007640190

2023-11-17T01:18:47.047-08:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_MERGE_PARTITION elapsed 2.527007ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error <nil>

2023-11-17T01:18:47.047-08:00 [Info] MergePartition: instance 9655805858047935938 merged to instance 6054010877007640190

2023-11-17T01:18:47.048-08:00 [Info] ShardRebalancer::destTokenToMergeOrReady Response reveived for merge of index instance: 9655805858047935938, realInst: 6054010877007640190

panic: MergePartition: Both proxy index Instance 7626474466556815012 and real index instance 12069286628324147430 are not found

goroutine 9289603 [running]:

github.com/couchbase/indexing/secondary/common.CrashOnError(...)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:509

github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).destTokenToMergeOrReady.func1({0xc0808779b0?, 0x1?}, 0xc01e422601?)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2637 +0x4f7

created by github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).destTokenToMergeOrReady in goroutine 9289602

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2597 +0x278

2023-11-17T01:18:48.384-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=c408804a5187a69c21cd88780d4a383e -isEnterprise=true --deploymentModel=default]