Testing on CB Cluster:   Changed the Plasma Log Segment Size to 1MB: saptarshisen@Saptarshis-MacBook-Pro indexing % git diff secondary/common/config.go diff --git a/secondary/common/config.go b/secondary/common/config.go index c5cdec67..6fdb796a 100644 --- a/secondary/common/config.go +++ b/secondary/common/config.go @@ -3126,10 +3126,5 @@ func (cv ConfigValue) Bool() bool { } func plasmaLogSegSize() int { - switch runtime.GOOS { - case "linux": - return 4 * 1024 * 1024 * 1024 - default: - return 512 * 1024 * 1024 - } + return 1 * 1024 * 1024 }    TestCases:   Delete a segment in datalog (With Patch):   saptarshisen@Saptarshis-MacBook-Pro mainIndex % ls -ltr total 212344 -rwxr-xr-x  1 saptarshisen  staff       64 Jun 30 14:37 config.json drwxr-xr-x  4 saptarshisen  staff      128 Jun 30 14:37 recovery -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000005.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000006.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000007.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000008.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000009.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000010.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000011.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000012.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000013.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000014.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000015.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:37 log.00000000000016.data -rwxr-xr-x  1 saptarshisen  staff  3846144 Jun 30 14:39 log.00000000000017.data -rwxr-xr-x  1 saptarshisen  staff     8192 Jun 30 14:39 header.data saptarshisen@Saptarshis-MacBook-Pro mainIndex % rm -f log.00000000000011.data    021-06-30T14:42:20.404-07:00 [Info] ServiceMgr::registerWithServer nodeuuid 46e686fe7ad138200a602836baf14164 2021-06-30T14:42:20.405-07:00 [Info] Indexer::initPartnInstance Initialized Partition:         Index: 2498162931070033083 Partition: PartitionId: 0 Endpoints: [:9105] 2021-06-30T14:42:20.406-07:00 [Info] Start shard recovery from /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/shards 2021-06-30T14:42:20.406-07:00 [Info] : Recover shard shard1 from metadata completed. 2021-06-30T14:42:20.407-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4be9840 IndexInstId 2498162931070033083 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_2498162931070033083_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_2498162931070033083_0.index/mainIndex fatal: detected missing log segment for fileId :12 (prev fileId :10) 2021-06-30T14:42:20.407-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 2498162931070033083 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable 2021-06-30T14:42:20.407-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst         InstId: 2498162931070033083         Defn: DefnId: 7502018329097337980 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0                 SecExprs: ([])                 Desc: []                 PartitionScheme: SINGLE                 HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false         State: INDEX_STATE_ACTIVE         RState: RebalActive         Stream: MAINT_STREAM         Version: 0         ReplicaId: 0         PartitionContainer: &{map[0:{0 0 [:9105]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9105]} 2021-06-30T14:42:20.418-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9105]   Delete  tail segment in datalog:   saptarshisen@Saptarshis-MacBook-Pro data % ls -l  n_2/data/@2i/default_\#primary_17210306779906375033_0.index/mainIndex/ total 205520 -rwxr-xr-x  1 saptarshisen  staff       64 Jun 30 14:51 config.json -rwxr-xr-x  1 saptarshisen  staff     8192 Jun 30 14:51 header.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000005.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000006.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000007.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000008.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000009.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000010.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000011.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000012.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000013.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000014.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000015.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:51 log.00000000000016.data -rwxr-xr-x  1 saptarshisen  staff  1286144 Jun 30 14:51 log.00000000000017.data drwxr-xr-x  4 saptarshisen  staff      128 Jun 30 14:51 recovery saptarshisen@Saptarshis-MacBook-Pro data % rm  n_2/data/@2i/default_\#primary_17210306779906375033_0.index/mainIndex/log.00000000000017.data  saptarshisen@Saptarshis-MacBook-Pro data %    2021-06-30T14:53:15.790-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4be9840 IndexInstId 17210306779906375033 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_17210306779906375033_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_17210306779906375033_0.index/mainIndex fatal: Corrupt lss log with tailOffset=143892480 > segmentOffset=142606336 - /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_17210306779906375033_0.index/mainIndex 2021-06-30T14:53:15.790-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 17210306779906375033 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable 2021-06-30T14:53:15.790-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst   Delete middle segment in datalog:   saptarshisen@Saptarshis-MacBook-Pro data % ls -ltr  n_1/data/@2i/default_\#primary_4253300863834747877_0.index/mainIndex/                   total 204560 -rwxr-xr-x  1 saptarshisen  staff       64 Jun 30 14:56 config.json drwxr-xr-x  4 saptarshisen  staff      128 Jun 30 14:56 recovery -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000005.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000006.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000007.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000008.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000009.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000010.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000011.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000012.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000013.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000014.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000015.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 14:56 log.00000000000016.data -rwxr-xr-x  1 saptarshisen  staff  2670592 Jun 30 14:56 log.00000000000017.data -rwxr-xr-x  1 saptarshisen  staff     8192 Jun 30 14:56 header.data saptarshisen@Saptarshis-MacBook-Pro data %  saptarshisen@Saptarshis-MacBook-Pro data % rm  n_1/data/@2i/default_\#primary_4253300863834747877_0.index/mainIndex/log.00000000000005.data   2021-06-30T15:14:08.219-07:00 [Info] : Recover shard shard2 from metadata completed. 2021-06-30T15:14:08.220-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4beaa60 IndexInstId 4253300863834747877 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_4253300863834747877_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_4253300863834747877_0.index/mainIndex fatal: detected missing log segments, expected range (5-17), found range (6-17) 2021-06-30T15:14:08.220-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 4253300863834747877 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable 2021-06-30T15:14:08.221-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst         InstId: 4253300863834747877         Defn: DefnId: 1721076629252085306 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0                 SecExprs: ([])                 Desc: []                 PartitionScheme: SINGLE                 HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false         State: INDEX_STATE_ACTIVE         RState: RebalActive         Stream: MAINT_STREAM         Version: 0         ReplicaId: 0         PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]} 2021-06-30T15:14:08.228-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9111] 2021-06-30T15:14:08.228-07:00 [Info] Indexer::forceCleanupIndexPartition 4253300863834747877 0 mark metadata as deleted 2021-06-30T15:14:08.228-07:00 [Info] ClustMgr:handleCleanupPartition     Truncate middle segment in datalog (Without Fix, observed panic in page.go):   saptarshisen@Saptarshis-MacBook-Pro mainIndex % ls -ltr total 208816 -rwxr-xr-x  1 saptarshisen  staff       64 Jun 30 15:16 config.json drwxr-xr-x  4 saptarshisen  staff      128 Jun 30 15:16 recovery -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000005.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000006.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000007.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000008.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000009.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000010.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000011.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000012.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000013.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000014.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000015.data -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 15:16 log.00000000000016.data -rwxr-xr-x  1 saptarshisen  staff  3551232 Jun 30 15:18 log.00000000000017.data -rwxr-xr-x  1 saptarshisen  staff     8192 Jun 30 15:18 header.data    truncate -s 4M log.00000000000008.data 2021-06-30T16:06:32.383-07:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = (["airline_57f6e99e-8f39-4915-8055-ac69a557c214"]) docid = (airline_57f6e99e-8f39-4915-8055-ac69a557c214) Index #primary, Bucket default, IndexInstId 17582325951252735427, PartitionId 0   goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0xc00040f450)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x6e created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x475   goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0xc000445e60)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x85 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x95   goroutine 143 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0xc000437300)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1232 +0x435 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1171 +0x129   /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/bufio/bufio.go:463 +0x4c github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(   /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/net/rpc/server.go:458 +0x9d github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc0014d4060, 0xc0014c4f10, 0x0, 0x0)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x56b github.com/couchbase/cbauth/revrpc.BabysitService(0xc001497f10, 0xc0014d4060, 0x0, 0x0, 0x0, 0x10)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x5c github.com/couchbase/cbauth/service.RegisterManager(0x5068958, 0xc000214e00, 0x0, 0x0, 0xc0014c4f80, 0x1)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xc5 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0xc000214e00)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:300 +0x152 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:204 +0x8d   /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:304 +0x8ae 2021-06-30T16:06:32.387-07:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = (["airline_38648e21-a8ac-4834-af27-22eccb6a139c"]) docid = (airline_38648e21-a8ac-4834-af27-22eccb6a139c) Index #primary, Bucket default, IndexInstId 17582325951252735427, PartitionId 0 panic: default/#primary/Mainstore#17582325951252735427:0 : fatal: fatal: segCount:3, EOF [recovered]         panic: default/#primary/Mainstore#17582325951252735427:0 : fatal: fatal: segCount:3, EOF   goroutine 282 [running]: github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1(0xc004bbfee8, 0xc004d4e580)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:614 +0x4b3 panic(0x4cd6360, 0xc0076260d0)         /Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/panic.go:965 +0x1b9 github.com/couchbase/plasma.(*Shard).raisePanic(...)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/shard.go:1047 github.com/couchbase/plasma.(*LSSCtx).raiseCorrupted(0xc004e0a280, 0x5043740, 0xc0076260d0)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/lssctx.go:181 +0x1ff github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc004e10600, 0xc0069fde18, 0x16)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/plasma.go:1488 +0x2b0 github.com/couchbase/plasma.(*pageWalker).readPageSegment(0xc004bbfb48)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page_walker.go:243 +0x2d7 github.com/couchbase/plasma.(*pageWalker).Next(0xc005fd9b48)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page_walker.go:227 +0xae github.com/couchbase/plasma.(*page).lookup(0xc007619340, 0xc004bbfb48, 0x3a94b280, 0x50563b0, 0xc0072dff50, 0x0, 0x3a94b201)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page.go:698 +0x85 github.com/couchbase/plasma.(*page).Lookup(0xc007619340, 0x3a94b280, 0x0, 0xd4c5b00)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page.go:676 +0x147 github.com/couchbase/plasma.(*Writer).Lookup(0xc005bea8c0, 0x3a94b280, 0x2c, 0x30, 0x0)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/writer.go:142 +0x10f github.com/couchbase/plasma.(*Writer).LookupKV(0xc005bea8c0, 0xc0073b0120, 0x2c, 0x30, 0x2, 0x10100000000, 0x0, 0x2, 0x1)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/mvcc.go:409 +0x10d github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertPrimaryIndex(0xc004d4e580, 0xc0073b0180, 0x30, 0x30, 0xc0073b0120, 0x2c, 0x30, 0x0, 0x0)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:770 +0x157 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0xc004d4e580, 0xc0073b0180, 0x30, 0x30, 0xc0073b0120, 0x2c, 0x30, 0x0, 0xc000148f00, 0xc0073a2240, ...)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:747 +0x273 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc004d4e580, 0x0)         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:627 +0x565 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters         /Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:3024 +0x53e Initializing write barrier = 8000   Truncate middle segment in datalog (With Fix):   saptarshisen@Saptarshis-MacBook-Pro mainIndex % ls -litr        total 218744 15289254 -rwxr-xr-x  1 saptarshisen  staff       64 Jun 30 17:11 config.json 15289256 drwxr-xr-x  4 saptarshisen  staff      128 Jun 30 17:11 recovery 15289364 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000005.data 15289383 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000006.data 15289385 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000007.data 15289386 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000008.data 15289405 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000009.data 15289429 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000010.data 15289441 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000011.data 15289459 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000012.data 15289467 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000013.data 15289476 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000014.data 15289480 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000015.data 15289490 -rwxr-xr-x  1 saptarshisen  staff  8388608 Jun 30 17:11 log.00000000000016.data 15289492 -rwxr-xr-x  1 saptarshisen  staff  7712768 Jun 30 17:11 log.00000000000017.data 15289255 -rwxr-xr-x  1 saptarshisen  staff     8192 Jun 30 17:11 header.data saptarshisen@Saptarshis-MacBook-Pro mainIndex % truncate -s 6M log.00000000000010.data  saptarshisen@Saptarshis-MacBook-Pro mainIndex %    021-06-30T17:15:25.124-07:00 [Info] Start shard recovery from /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/shards 2021-06-30T17:15:25.126-07:00 [Info] : Recover shard shard2 from metadata completed. 2021-06-30T17:15:25.126-07:00 [Info] : Recover shard shard3 from metadata completed. 2021-06-30T17:15:25.127-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea9a0 IndexInstId 3744260744496349541 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_3744260744496349541_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_3744260744496349541_0.index/mainIndex fatal: Corrupt lss log, detected invalid segment size for fileId 11, expected :8388608, found :6291456 2021-06-30T17:15:25.127-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 3744260744496349541 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable 2021-06-30T17:15:25.127-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst         InstId: 3744260744496349541         Defn: DefnId: 4254289076574466823 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0                 SecExprs: ([])                 Desc: []                 PartitionScheme: SINGLE                 HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false         State: INDEX_STATE_ACTIVE         RState: RebalActive         Stream: MAINT_STREAM         Version: 0         ReplicaId: 0         PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]}   Truncate middle segment in datalog (With Fix):   2021-06-30T19:54:40.284-07:00 [Info] Indexer::forceCleanupIndexPartition 480511311607607629 0 actually delete metadata 2021-06-30T19:54:40.284-07:00 [Info] ClustMgr:handleCleanupPartition         Message: MsgCleanupPartition         Type: 67         Index defn Id: 13355977755905551694 2021-06-30T19:54:40.212-07:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-06-30T19:54:40.212-07:00 [Info] Indexer::local storage mode plasma 2021-06-30T19:54:40.212-07:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-06-30T19:54:40.212-07:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-06-30T19:54:40.265-07:00 [Info] schedIndexCreator: intialized. 2021-06-30T19:54:40.265-07:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-06-30T19:54:40.265-07:00 [Info] RebalanceMgr::initService Init 2021-06-30T19:54:40.265-07:00 [Info] ServiceMgr::registerWithServer nodeuuid e97b221f985c590c330d6a674ef0686d 2021-06-30T19:54:40.265-07:00 [Info] Indexer::initPartnInstance Initialized Partition:         Index: 480511311607607629 Partition: PartitionId: 0 Endpoints: [:9111] 2021-06-30T19:54:40.267-07:00 [Info] Start shard recovery from /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/shards 2021-06-30T19:54:40.267-07:00 [Info] : Recover shard shard4 from metadata completed. 2021-06-30T19:54:40.267-07:00 [Info] : Recover shard shard5 from metadata completed. 2021-06-30T19:54:40.268-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea8e0 IndexInstId 480511311607607629 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_480511311607607629_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_480511311607607629_0.index/mainIndex fatal: Corrupt lss log with tailOffset=11149312 >segmentOffset=9437184 2021-06-30T19:54:40.268-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 480511311607607629 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable 2021-06-30T19:54:40.268-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst         InstId: 480511311607607629         Defn: DefnId: 13355977755905551694 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0                 SecExprs: ([])                 Desc: []                 PartitionScheme: SINGLE                 HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false         State: INDEX_STATE_ACTIVE         RState: RebalActive         Stream: MAINT_STREAM         Version: 0         ReplicaId: 0         PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]} 2021-06-30T19:54:40.276-07:00 [Info] ServiceMgr::GetTaskList [] 2021-06-30T19:54:40.276-07:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 0] []} 2021-06-30T19:54:40.278-07:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-06-30T19:54:40.278-07:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 0] [e97b221f985c590c330d6a674ef0686d] true []} 2021-06-30T19:54:40.279-07:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 0]   bc 1.06 Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc. This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'.  8388608 + (1024 * 1024) 9437184   Delete segment in Recovery Log (With Fix):   saptarshisen@Saptarshis-MacBook-Pro data % ls -lih n_1/data/@2i/default_\#primary_5114813154892359787_0.index/mainIndex/recovery total 5456 15383552 -rwxr-xr-x  1 saptarshisen  staff   8.0K Jun 30 20:42 header.data 15383553 -rwxr-xr-x  1 saptarshisen  staff   1.0M Jun 30 20:32 log.00000000000000.data 15411309 -rwxr-xr-x  1 saptarshisen  staff   1.0M Jun 30 20:42 log.00000000000001.data 15419885 -rwxr-xr-x  1 saptarshisen  staff   648K Jun 30 20:42 log.00000000000002.data saptarshisen@Saptarshis-MacBook-Pro data %           saptarshisen@Saptarshis-MacBook-Pro data % mv n_1/data/@2i/default_\#primary_5114813154892359787_0.index/mainIndex/recovery/log.00000000000000.data /tmp/log.00000000000000.data   2021-06-30T21:31:31.731-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea8e0 IndexInstId 5114813154892359787 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_5114813154892359787_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_5114813154892359787_0.index/mainIndex/recovery fatal: Corrupt lss log, superblock log segments not present, found fileIds :1 - 2, superblock fileIds :0 - 2 2021-06-30T21:31:31.731-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 5114813154892359787 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable 2021-06-30T21:31:31.731-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst         InstId: 5114813154892359787         Defn: DefnId: 3581653491356785672 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0                 SecExprs: ([])                 Desc: []                 PartitionScheme: SINGLE                 HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false         State: INDEX_STATE_ACTIVE         RState: RebalActive         Stream: MAINT_STREAM         Version: 0         ReplicaId: 0         PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]} 2021-06-30T21:31:31.742-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9111] 2021-06-30T21:31:31.742-07:00 [Info] Indexer::forceCleanupIndexPartition 5114813154892359787 0 mark metadata as deleted 2021-06-30T21:31:31.742-07:00 [Info] ClustMgr:handleCleanupPartition         Message: MsgCleanupPartition         Type: 67         Index defn Id: 3581653491356785672         Index inst Id: 5114813154892359787         Index partition Id: 0         Index replica Id: 0         Update Status Only: true 2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 3581653491356785672 instance id 5114813154892359787 real instance id 0 partitions [0] 2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 3581653491356785672 instance 5114813154892359787 partitions [0] 2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 3581653491356785672 instance id 5114813154892359787 2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance.  Delete index 3581653491356785672   2021-06-30T21:40:35.986-07:00 [Info] LSS /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex(shard5) : LSSCtx.cleanup: Done cleanup 2021-06-30T21:40:35.986-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea8e0 IndexInstId 7889820684348130190 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex/recovery fatal: Corrupt lss log with tailOffset=1978368 >segmentOffset=1978367 2021-06-30T21:40:35.986-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 7889820684348130190 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable 2021-06-30T21:40:35.986-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst         InstId: 7889820684348130190         Defn: DefnId: 4795002632039732232 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0                 SecExprs: ([])                 Desc: []                 PartitionScheme: SINGLE                 HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false         State: INDEX_STATE_ACTIVE         RState: RebalActive         Stream: MAINT_STREAM     saptarshisen@Saptarshis-MacBook-Pro data % ls -li n_2/data/@2i/default_\#primary_7889820684348130190_0.index/mainIndex/recovery  total 3880 15447438 -rwxr-xr-x  1 saptarshisen  staff     8192 Jun 30 21:36 header.data 15447439 -rwxr-xr-x  1 saptarshisen  staff  1048576 Jun 30 21:36 log.00000000000000.data 15447592 -rwxr-xr-x  1 saptarshisen  staff   929792 Jun 30 21:36 log.00000000000001.data saptarshisen@Saptarshis-MacBook-Pro data % truncate -s 929791 n_2/data/@2i/default_\#primary_7889820684348130190_0.index/mainIndex/recovery/log.00000000000001.data         Version: 0         ReplicaId: 0         PartitionContainer: &{map[0:{0 0 [:9117]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9117]} 2021-06-30T21:40:35.990-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9117] 2021-06-30T21:40:35.990-07:00 [Info] Indexer::forceCleanupIndexPartition 7889820684348130190 0 mark metadata as deleted         Update Status Only: true 2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 4795002632039732232 instance id 7889820684348130190 real instance id 0 partitions [0] 2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 4795002632039732232 instance 7889820684348130190 partitions [0] 2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 4795002632039732232 instance id 7889820684348130190 2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance.  Delete index 4795002632039732232 2021-06-30T21:40:35.991-07:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CLEANUP_PARTITION elapsed 1.117917ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 error 2021-06-30T21:40:35.991-07:00 [Info] Indexer::forceCleanupIndexPartition Cleaning up data files for 7889820684348130190 0 Destory instances matching prefix /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index in /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i ... 2021-06-30T21:40:35.991-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : destroying instance /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex ... 2021-06-30T21:40:35.995-07:00 [Info] ServiceMgr::updateNodeList Updated Node List [46e686fe7ad138200a602836baf14164 e97b221f985c590c330d6a674ef0686d ef751350af236e270f66e34211da4441 0c3eec0fe078e88f1267f301fc92842b] 2021-06-30T21:40:35.995-07:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 1] [46e686fe7ad138200a602836baf14164 e97b221f985c590c330d6a674ef0686d ef751350af236e270f66e34211da4441 0c3eec0fe078e88f1267f301fc92842b] true []} 2021-06-30T21:40:35.995-07:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 1] []} 2021-06-30T21:40:35.998-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : removed plasmaId 1 for instance /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex ... 2021-06-30T21:40:35.999-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : metadata saved successfully 2021-06-30T21:40:35.999-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : instance /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex sucessfully destroyed All instances matching prefix /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index destroyed 2021-06-30T21:40:35.999-07:00 [Info] Indexer::forceCleanupIndexPartition 7889820684348130190 0 Cleanup partition in-memory data structure 2021-06-30T21:40:35.999-07:00 [Info] Indexer::forceCleanupIndexPartition 7889820684348130190 0 actually delete metadata 2021-06-30T21:40:35.999-07:00 [Info] ClustMgr:handleCleanupPartition         Message: MsgCleanupPartition         Type: 67         Index defn Id: 4795002632039732232         Index inst Id: 7889820684348130190         Index partition Id: 0         Index replica Id: 0         Update Status Only: false 2021-06-30T21:40:35.999-07:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 4795002632039732232 instance id 7889820684348130190 real instance id 0 partitions [0] 2021-06-30T21:40:35.999-07:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 4795002632039732232 instance 7889820684348130190 partitions [0]   Updated Patch:   2021-07-01T12:44:40.449-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4beaa80 IndexInstId 18417244362498523011 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_18417244362498523011_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_18417244362498523011_0.index/mainIndex fatal: Corrupt lss log with end segmentOffset=83701759 < superblock tailOffset=83701760   -rwxr-xr-x  1 saptarshisen  staff  1048576 Jul  1 12:32 log.00000000000078.data drwxr-xr-x  5 saptarshisen  staff      160 Jul  1 12:32 recovery -rwxr-xr-x  1 saptarshisen  staff  864256 Jul  1 12:32 log.00000000000079.data -rwxr-xr-x  1 saptarshisen  staff    8192 Jul  1 12:32 header.data saptarshisen@Saptarshis-MacBook-Pro mainIndex % truncate -s 864255 log.00000000000079.data   Unit Tests:   saptarshisen@Saptarshis-MacBook-Pro plasma % go test -v -run TestLogMissingAndTruncatedSegment Initializing write barrier = 8000 === RUN  TestLogMissingAndTruncatedSegments (expected) error reading log file :test.TestLogMissingAndTruncatedSegments fatal: Corrupt lss log with start segmentOffset=1048576 >superblock headOffset=0 (expected) error reading log file :test.TestLogMissingAndTruncatedSegments fatal: Corrupt lss log with end segmentOffset=19922944 CREATE PRIMARY INDEX ON `default` using GSI; {     "requestID": "c4c77700-46d6-4064-874d-26695db88214",     "signature": null,     "results": [     ],     "status": "success",     "metrics": {         "elapsedTime": "8.573391324s",         "executionTime": "8.573337606s",         "resultCount": 0,         "resultSize": 0,         "serviceLoad": 2     } }