As seen on svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com
2023-07-11T13:10:30.785+00:00 [Info] StorageMgr::openSnapshot IndexInst:8473785027485727187 Partition:7 Attempting to open snapshot (SnapshotInfo: count:613 committed:false)
|
2023-07-11T13:10:30.789+00:00 [Info] StorageMgr::openSnapshot IndexInst:8473785027485727187 Partition:8 Attempting to open snapshot (SnapshotInfo: count:661 committed:false)
|
2023-07-11T13:10:30.789+00:00 [Info] ScanCoordinator::handleAddIndexInstance &{0xc0004205a0 ScanCoordinator {8473785027485727187 {3986530505751148185 idx11_tSsjeVA plasma default3 b4f99bf45ece591030e7d9373fa7d84d false [(all (array flatten_keys(((`r`.`ratings`).`Rooms`), ((`r`.`ratings`).`Cleanliness`)) for `r` in `reviews` end)) (all (array flatten_keys(((`r`.`ratings`).`Rooms`), ((`r`.`ratings`).`Cleanliness`)) for `r` in `reviews` end)) `email` `free_parking`] N1QL KEY [false false false false] false true [] true true 1 [(meta().`id`)] false 0 {true 1 0 0} scope_0 coll_0 8 8 true false true 0 0 0 0 0 0 0 0 [] [] 0 0} 4 0 1 0xc0655f5dc0 [] 0 0 false plasma 0} map[4:{{4 0 [:9105]} 0xc06311e7d0} 5:{{5 0 [:9105]} 0xc061b84e70} 6:{{6 0 [:9105]} 0xc0607ba2b0} 7:{{7 0 [:9105]} 0xc063977500} 8:{{8 0 [:9105]} 0xc061b854c0}] 0xc064380000 0xc037c8c600}
|
2023-07-11T13:10:30.789+00:00 [Info] Indexer::initPartnInstance Initialized Partition:
|
Index: 7502033646761757378 Partition: PartitionId: 0 Endpoints: [:9105]
|
2023-07-11T13:10:30.789+00:00 [Error] plasmaSlice:NewplasmaSlice Id 0x10992c0 IndexInstId 7502033646761757378 fatal error occured: Unable to initialize /var/cb/data/@2i/default7_idx12_XZMSU2BOQB_idxprefix_7502033646761757378_0.index/mainIndex, err = fatal: Fail to find shard for shared instance /var/cb/data/@2i/default7_idx12_XZMSU2BOQB_idxprefix_7502033646761757378_0.index/mainIndex due to instance not present on disk
|
2023-07-11T13:10:30.789+00:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 7502033646761757378 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
|
2023-07-11T13:10:30.790+00:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
|
InstId: 7502033646761757378
|
Defn: DefnId: 17342978310820429283 Name: idx12_XZMSU2BOQB_idxprefix Using: plasma Bucket: default7 Scope/Id: scope_1/9 Collection/Id: coll_0/9 IsPrimary: false NumReplica: 1 InstVersion: 0
|
SecExprs: <ud>([`name` `phone` `type`])</ud>
|
Desc: [true false false]
|
IndexMissingLeadingKey: true
|
IsPartnKeyDocId: false
|
PartitionScheme: SINGLE
|
HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false
|
State: INDEX_STATE_CREATED
|
RState: RebalActive
|
Stream: NIL_STREAM
|
Version: 0
|
ReplicaId: 1
|
PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} partnDefn {0 0 [:9105]}
|
2023-07-11T13:10:30.796+00:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9105]
|
cbcollect ->
https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-d-node-001.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-d-node-002.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-d-node-003.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-qi-node-004.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/SysTest_11Jul_Slow_Indexing/collectinfo-2023-07-12T040332-ns_1%40svc-qi-node-005.r45djlf-eb-k-m7.sandbox.nonprod-project-avengers.com.zip
|
A similar storage corruption error was seen on a 7.1.4-3632 with an AWS run as well. Not really sure if they are both similar.
https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-001.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-002.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-003.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-006.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-qi-node-004.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-qi-node-005.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
|
https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-qi-node-007.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
|
Improvement
Critical