[System Test] Indexer panic - mainIndex found with mismatching alternateId

023-10-20T05:25:30.271-07:00 [Info] StorageMgr::updateIndexSnapMapForIndex IndexInst 5923803431981465048 Partitions [0]

2023-10-20T05:25:30.271-07:00 [Info] StorageMgr::openSnapshot IndexInst:5923803431981465048 Partition:0 No Snapshot Found.

2023-10-20T05:25:30.271-07:00 [Info] StorageMgr::updateIndexSnapMapForIndex IndexInst 5923803431981465048 Adding Nil Snapshot.

2023-10-20T05:25:30.271-07:00 [Info] ScanCoordinator::handleAddIndexInstance &{0xc000182840 ScanCoordinator {5923803431981465048 {8991753871806477285 idx13_CayqIaU plasma bucket2 6bcea1f07a1e27442dfb127d5df17215 false [`city` `phone`] N1QL SINGLE   [false false] true false [] false false 3 [] false 0 {true 3 0 0} scope_0 coll_11 9 15 false true false 0 0 0 0 0 0 0 0 [] [] 0 0 0 map[] map[0:[16993787915537584520-3-0 16993787915537584520-3-1]]} 0 0 0 0xc00b2ea210  [] 0 3 false plasma  0} map[0:{{0 0 [:9105] [7086915666389875762 8694078220371084489]} 0xc00794ef10}] 0xc00b2fc000 0xc00af3eea0}

2023-10-20T05:25:30.271-07:00 [Info] Indexer::initPartnInstance Initialized Partition: 

	 Index: 2961634984762461950 Partition: PartitionId: 1 Endpoints: [:9105] , shardIds: map[]

2023-10-20T05:25:30.271-07:00 [Error] Indexer::initPartnInstance Error creating slice Unable to initialize /data/@2i/bucket1_idx6_8BHsx_2961634984762461950_1.index/mainIndex, err = Instance /data/@2i/bucket1_idx6_8BHsx_2961634984762461950_1.index/mainIndex found with mismatching alternateId, instance AltId:, shard AltdId:16993787915537584520-3-0. Abort.

2023-10-20T05:25:30.271-07:00 [Fatal] Indexer::Unable to Bootstrap Indexer from Persisted Metadata Error creating slice Unable to initialize /data/@2i/bucket1_idx6_8BHsx_2961634984762461950_1.index/mainIndex, err = Instance /data/@2i/bucket1_idx6_8BHsx_2961634984762461950_1.index/mainIndex found with mismatching alternateId, instance AltId:, shard AltdId:16993787915537584520-3-0

panic: Error creating slice Unable to initialize /data/@2i/bucket1_idx6_8BHsx_2961634984762461950_1.index/mainIndex, err = Instance /data/@2i/bucket1_idx6_8BHsx_2961634984762461950_1.index/mainIndex found with mismatching alternateId, instance AltId:, shard AltdId:16993787915537584520-3-0

goroutine 1 [running]:

github.com/couchbase/indexing/secondary/common.CrashOnError(...)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:509

github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone(0xc00007f800?, {0x1f42de0?, 0xc0074ae510?})

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8330 +0xa70

github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0xc00007f800, {0x1f42de0, 0xc0074ae510})

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1568 +0xf5b

github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc00007f800)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1170 +0x365

github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x2e0c250?)

	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:593 +0x2d2c

main.main()