Testing on CB Cluster:
Changed the Plasma Log Segment Size to 1MB:
saptarshisen@Saptarshis-MacBook-Pro indexing % git diff secondary/common/config.go
diff --git a/secondary/common/config.go b/secondary/common/config.go
index c5cdec67..6fdb796a 100644
--- a/secondary/common/config.go
+++ b/secondary/common/config.go
@@ -3126,10 +3126,5 @@ func (cv ConfigValue) Bool() bool {
}
func plasmaLogSegSize() int {
- switch runtime.GOOS {
- case "linux":
- return 4 * 1024 * 1024 * 1024
- default:
- return 512 * 1024 * 1024
- }
+ return 1 * 1024 * 1024
}
TestCases:
Delete a segment in datalog (With Patch):
saptarshisen@Saptarshis-MacBook-Pro mainIndex % ls -ltr
total 212344
-rwxr-xr-x 1 saptarshisen staff 64 Jun 30 14:37 config.json
drwxr-xr-x 4 saptarshisen staff 128 Jun 30 14:37 recovery
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000005.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000006.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000007.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000008.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000009.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000010.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000011.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000012.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000013.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000014.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000015.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:37 log.00000000000016.data
-rwxr-xr-x 1 saptarshisen staff 3846144 Jun 30 14:39 log.00000000000017.data
-rwxr-xr-x 1 saptarshisen staff 8192 Jun 30 14:39 header.data
saptarshisen@Saptarshis-MacBook-Pro mainIndex % rm -f log.00000000000011.data
021-06-30T14:42:20.404-07:00 [Info] ServiceMgr::registerWithServer nodeuuid 46e686fe7ad138200a602836baf14164
2021-06-30T14:42:20.405-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
Index: 2498162931070033083 Partition: PartitionId: 0 Endpoints: [:9105]
2021-06-30T14:42:20.406-07:00 [Info] Start shard recovery from /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/shards
2021-06-30T14:42:20.406-07:00 [Info] : Recover shard shard1 from metadata completed.
2021-06-30T14:42:20.407-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4be9840 IndexInstId 2498162931070033083 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_2498162931070033083_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_2498162931070033083_0.index/mainIndex fatal: detected missing log segment for fileId :12 (prev fileId :10)
2021-06-30T14:42:20.407-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 2498162931070033083 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-06-30T14:42:20.407-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
InstId: 2498162931070033083
Defn: DefnId: 7502018329097337980 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0
SecExprs: ([])
Desc: []
PartitionScheme: SINGLE
HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false
State: INDEX_STATE_ACTIVE
RState: RebalActive
Stream: MAINT_STREAM
Version: 0
ReplicaId: 0
PartitionContainer: &{map[0:{0 0 [:9105]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9105]}
2021-06-30T14:42:20.418-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9105]
Delete tail segment in datalog:
saptarshisen@Saptarshis-MacBook-Pro data % ls -l n_2/data/@2i/default_\#primary_17210306779906375033_0.index/mainIndex/
total 205520
-rwxr-xr-x 1 saptarshisen staff 64 Jun 30 14:51 config.json
-rwxr-xr-x 1 saptarshisen staff 8192 Jun 30 14:51 header.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000005.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000006.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000007.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000008.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000009.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000010.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000011.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000012.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000013.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000014.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000015.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:51 log.00000000000016.data
-rwxr-xr-x 1 saptarshisen staff 1286144 Jun 30 14:51 log.00000000000017.data
drwxr-xr-x 4 saptarshisen staff 128 Jun 30 14:51 recovery
saptarshisen@Saptarshis-MacBook-Pro data % rm n_2/data/@2i/default_\#primary_17210306779906375033_0.index/mainIndex/log.00000000000017.data
saptarshisen@Saptarshis-MacBook-Pro data %
2021-06-30T14:53:15.790-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4be9840 IndexInstId 17210306779906375033 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_17210306779906375033_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_17210306779906375033_0.index/mainIndex fatal: Corrupt lss log with tailOffset=143892480 > segmentOffset=142606336 - /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_17210306779906375033_0.index/mainIndex
2021-06-30T14:53:15.790-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 17210306779906375033 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-06-30T14:53:15.790-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
Delete middle segment in datalog:
saptarshisen@Saptarshis-MacBook-Pro data % ls -ltr n_1/data/@2i/default_\#primary_4253300863834747877_0.index/mainIndex/
total 204560
-rwxr-xr-x 1 saptarshisen staff 64 Jun 30 14:56 config.json
drwxr-xr-x 4 saptarshisen staff 128 Jun 30 14:56 recovery
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000005.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000006.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000007.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000008.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000009.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000010.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000011.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000012.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000013.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000014.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000015.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 14:56 log.00000000000016.data
-rwxr-xr-x 1 saptarshisen staff 2670592 Jun 30 14:56 log.00000000000017.data
-rwxr-xr-x 1 saptarshisen staff 8192 Jun 30 14:56 header.data
saptarshisen@Saptarshis-MacBook-Pro data %
saptarshisen@Saptarshis-MacBook-Pro data % rm n_1/data/@2i/default_\#primary_4253300863834747877_0.index/mainIndex/log.00000000000005.data
2021-06-30T15:14:08.219-07:00 [Info] : Recover shard shard2 from metadata completed.
2021-06-30T15:14:08.220-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4beaa60 IndexInstId 4253300863834747877 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_4253300863834747877_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_4253300863834747877_0.index/mainIndex fatal: detected missing log segments, expected range (5-17), found range (6-17)
2021-06-30T15:14:08.220-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 4253300863834747877 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-06-30T15:14:08.221-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
InstId: 4253300863834747877
Defn: DefnId: 1721076629252085306 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0
SecExprs: ([])
Desc: []
PartitionScheme: SINGLE
HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false
State: INDEX_STATE_ACTIVE
RState: RebalActive
Stream: MAINT_STREAM
Version: 0
ReplicaId: 0
PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]}
2021-06-30T15:14:08.228-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9111]
2021-06-30T15:14:08.228-07:00 [Info] Indexer::forceCleanupIndexPartition 4253300863834747877 0 mark metadata as deleted
2021-06-30T15:14:08.228-07:00 [Info] ClustMgr:handleCleanupPartition
Truncate middle segment in datalog (Without Fix, observed panic in page.go):
saptarshisen@Saptarshis-MacBook-Pro mainIndex % ls -ltr
total 208816
-rwxr-xr-x 1 saptarshisen staff 64 Jun 30 15:16 config.json
drwxr-xr-x 4 saptarshisen staff 128 Jun 30 15:16 recovery
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000005.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000006.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000007.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000008.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000009.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000010.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000011.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000012.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000013.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000014.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000015.data
-rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 15:16 log.00000000000016.data
-rwxr-xr-x 1 saptarshisen staff 3551232 Jun 30 15:18 log.00000000000017.data
-rwxr-xr-x 1 saptarshisen staff 8192 Jun 30 15:18 header.data
truncate -s 4M log.00000000000008.data
2021-06-30T16:06:32.383-07:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = (["airline_57f6e99e-8f39-4915-8055-ac69a557c214"]) docid = (airline_57f6e99e-8f39-4915-8055-ac69a557c214) Index #primary, Bucket default, IndexInstId 17582325951252735427, PartitionId 0
goroutine 48 [chan receive]:
github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0xc00040f450)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x6e
created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x475
goroutine 49 [chan receive]:
github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0xc000445e60)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x85
created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x95
goroutine 143 [select]:
github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0xc000437300)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1232 +0x435
created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1171 +0x129
/Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/bufio/bufio.go:463 +0x4c
github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(
/Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/net/rpc/server.go:458 +0x9d
github.com/couchbase/cbauth/revrpc.(*Service).Run(0xc0014d4060, 0xc0014c4f10, 0x0, 0x0)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x56b
github.com/couchbase/cbauth/revrpc.BabysitService(0xc001497f10, 0xc0014d4060, 0x0, 0x0, 0x0, 0x10)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x5c
github.com/couchbase/cbauth/service.RegisterManager(0x5068958, 0xc000214e00, 0x0, 0x0, 0xc0014c4f80, 0x1)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xc5
github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0xc000214e00)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:300 +0x152
created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:204 +0x8d
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:304 +0x8ae
2021-06-30T16:06:32.387-07:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = (["airline_38648e21-a8ac-4834-af27-22eccb6a139c"]) docid = (airline_38648e21-a8ac-4834-af27-22eccb6a139c) Index #primary, Bucket default, IndexInstId 17582325951252735427, PartitionId 0
panic: default/#primary/Mainstore#17582325951252735427:0 : fatal: fatal: segCount:3, EOF [recovered]
panic: default/#primary/Mainstore#17582325951252735427:0 : fatal: fatal: segCount:3, EOF
goroutine 282 [running]:
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1(0xc004bbfee8, 0xc004d4e580)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:614 +0x4b3
panic(0x4cd6360, 0xc0076260d0)
/Users/saptarshisen/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/panic.go:965 +0x1b9
github.com/couchbase/plasma.(*Shard).raisePanic(...)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/shard.go:1047
github.com/couchbase/plasma.(*LSSCtx).raiseCorrupted(0xc004e0a280, 0x5043740, 0xc0076260d0)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/lssctx.go:181 +0x1ff
github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc004e10600, 0xc0069fde18, 0x16)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/plasma.go:1488 +0x2b0
github.com/couchbase/plasma.(*pageWalker).readPageSegment(0xc004bbfb48)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page_walker.go:243 +0x2d7
github.com/couchbase/plasma.(*pageWalker).Next(0xc005fd9b48)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page_walker.go:227 +0xae
github.com/couchbase/plasma.(*page).lookup(0xc007619340, 0xc004bbfb48, 0x3a94b280, 0x50563b0, 0xc0072dff50, 0x0, 0x3a94b201)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page.go:698 +0x85
github.com/couchbase/plasma.(*page).Lookup(0xc007619340, 0x3a94b280, 0x0, 0xd4c5b00)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/page.go:676 +0x147
github.com/couchbase/plasma.(*Writer).Lookup(0xc005bea8c0, 0x3a94b280, 0x2c, 0x30, 0x0)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/writer.go:142 +0x10f
github.com/couchbase/plasma.(*Writer).LookupKV(0xc005bea8c0, 0xc0073b0120, 0x2c, 0x30, 0x2, 0x10100000000, 0x0, 0x2, 0x1)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/plasma/mvcc.go:409 +0x10d
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertPrimaryIndex(0xc004d4e580, 0xc0073b0180, 0x30, 0x30, 0xc0073b0120, 0x2c, 0x30, 0x0, 0x0)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:770 +0x157
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0xc004d4e580, 0xc0073b0180, 0x30, 0x30, 0xc0073b0120, 0x2c, 0x30, 0x0, 0xc000148f00, 0xc0073a2240, ...)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:747 +0x273
github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc004d4e580, 0x0)
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:627 +0x565
created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
/Users/saptarshisen/ws/couchbase_svr/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:3024 +0x53e
Initializing write barrier = 8000
Truncate middle segment in datalog (With Fix):
saptarshisen@Saptarshis-MacBook-Pro mainIndex % ls -litr
total 218744
15289254 -rwxr-xr-x 1 saptarshisen staff 64 Jun 30 17:11 config.json
15289256 drwxr-xr-x 4 saptarshisen staff 128 Jun 30 17:11 recovery
15289364 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000005.data
15289383 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000006.data
15289385 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000007.data
15289386 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000008.data
15289405 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000009.data
15289429 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000010.data
15289441 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000011.data
15289459 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000012.data
15289467 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000013.data
15289476 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000014.data
15289480 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000015.data
15289490 -rwxr-xr-x 1 saptarshisen staff 8388608 Jun 30 17:11 log.00000000000016.data
15289492 -rwxr-xr-x 1 saptarshisen staff 7712768 Jun 30 17:11 log.00000000000017.data
15289255 -rwxr-xr-x 1 saptarshisen staff 8192 Jun 30 17:11 header.data
saptarshisen@Saptarshis-MacBook-Pro mainIndex % truncate -s 6M log.00000000000010.data
saptarshisen@Saptarshis-MacBook-Pro mainIndex %
021-06-30T17:15:25.124-07:00 [Info] Start shard recovery from /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/shards
2021-06-30T17:15:25.126-07:00 [Info] : Recover shard shard2 from metadata completed.
2021-06-30T17:15:25.126-07:00 [Info] : Recover shard shard3 from metadata completed.
2021-06-30T17:15:25.127-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea9a0 IndexInstId 3744260744496349541 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_3744260744496349541_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_3744260744496349541_0.index/mainIndex fatal: Corrupt lss log, detected invalid segment size for fileId 11, expected :8388608, found :6291456
2021-06-30T17:15:25.127-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 3744260744496349541 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-06-30T17:15:25.127-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
InstId: 3744260744496349541
Defn: DefnId: 4254289076574466823 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0
SecExprs: ([])
Desc: []
PartitionScheme: SINGLE
HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false
State: INDEX_STATE_ACTIVE
RState: RebalActive
Stream: MAINT_STREAM
Version: 0
ReplicaId: 0
PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]}
Truncate middle segment in datalog (With Fix):
2021-06-30T19:54:40.284-07:00 [Info] Indexer::forceCleanupIndexPartition 480511311607607629 0 actually delete metadata
2021-06-30T19:54:40.284-07:00 [Info] ClustMgr:handleCleanupPartition
Message: MsgCleanupPartition
Type: 67
Index defn Id: 13355977755905551694
2021-06-30T19:54:40.212-07:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ...
2021-06-30T19:54:40.212-07:00 [Info] Indexer::local storage mode plasma
2021-06-30T19:54:40.212-07:00 [Info] DDLServiceMgr: starting buildTokenCleaner ...
2021-06-30T19:54:40.212-07:00 [Info] DDLServiceMgr: starting delTokenCleaner ...
2021-06-30T19:54:40.265-07:00 [Info] schedIndexCreator: intialized.
2021-06-30T19:54:40.265-07:00 [Info] RebalanceMgr::NewRebalanceMgr false
2021-06-30T19:54:40.265-07:00 [Info] RebalanceMgr::initService Init
2021-06-30T19:54:40.265-07:00 [Info] ServiceMgr::registerWithServer nodeuuid e97b221f985c590c330d6a674ef0686d
2021-06-30T19:54:40.265-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
Index: 480511311607607629 Partition: PartitionId: 0 Endpoints: [:9111]
2021-06-30T19:54:40.267-07:00 [Info] Start shard recovery from /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/shards
2021-06-30T19:54:40.267-07:00 [Info] : Recover shard shard4 from metadata completed.
2021-06-30T19:54:40.267-07:00 [Info] : Recover shard shard5 from metadata completed.
2021-06-30T19:54:40.268-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea8e0 IndexInstId 480511311607607629 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_480511311607607629_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_480511311607607629_0.index/mainIndex fatal: Corrupt lss log with tailOffset=11149312 >segmentOffset=9437184
2021-06-30T19:54:40.268-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 480511311607607629 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-06-30T19:54:40.268-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
InstId: 480511311607607629
Defn: DefnId: 13355977755905551694 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0
SecExprs: ([])
Desc: []
PartitionScheme: SINGLE
HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false
State: INDEX_STATE_ACTIVE
RState: RebalActive
Stream: MAINT_STREAM
Version: 0
ReplicaId: 0
PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]}
2021-06-30T19:54:40.276-07:00 [Info] ServiceMgr::GetTaskList []
2021-06-30T19:54:40.276-07:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 0] []}
2021-06-30T19:54:40.278-07:00 [Info] ServiceMgr::GetCurrentTopology []
2021-06-30T19:54:40.278-07:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 0] [e97b221f985c590c330d6a674ef0686d] true []}
2021-06-30T19:54:40.279-07:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 0]
bc 1.06
Copyright 1991-1994, 1997, 1998, 2000 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.
8388608 + (1024 * 1024)
9437184
Delete segment in Recovery Log (With Fix):
saptarshisen@Saptarshis-MacBook-Pro data % ls -lih n_1/data/@2i/default_\#primary_5114813154892359787_0.index/mainIndex/recovery
total 5456
15383552 -rwxr-xr-x 1 saptarshisen staff 8.0K Jun 30 20:42 header.data
15383553 -rwxr-xr-x 1 saptarshisen staff 1.0M Jun 30 20:32 log.00000000000000.data
15411309 -rwxr-xr-x 1 saptarshisen staff 1.0M Jun 30 20:42 log.00000000000001.data
15419885 -rwxr-xr-x 1 saptarshisen staff 648K Jun 30 20:42 log.00000000000002.data
saptarshisen@Saptarshis-MacBook-Pro data %
saptarshisen@Saptarshis-MacBook-Pro data % mv n_1/data/@2i/default_\#primary_5114813154892359787_0.index/mainIndex/recovery/log.00000000000000.data /tmp/log.00000000000000.data
2021-06-30T21:31:31.731-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea8e0 IndexInstId 5114813154892359787 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_5114813154892359787_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_1/data/@2i/default_#primary_5114813154892359787_0.index/mainIndex/recovery fatal: Corrupt lss log, superblock log segments not present, found fileIds :1 - 2, superblock fileIds :0 - 2
2021-06-30T21:31:31.731-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 5114813154892359787 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-06-30T21:31:31.731-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
InstId: 5114813154892359787
Defn: DefnId: 3581653491356785672 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0
SecExprs: ([])
Desc: []
PartitionScheme: SINGLE
HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false
State: INDEX_STATE_ACTIVE
RState: RebalActive
Stream: MAINT_STREAM
Version: 0
ReplicaId: 0
PartitionContainer: &{map[0:{0 0 [:9111]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9111]}
2021-06-30T21:31:31.742-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9111]
2021-06-30T21:31:31.742-07:00 [Info] Indexer::forceCleanupIndexPartition 5114813154892359787 0 mark metadata as deleted
2021-06-30T21:31:31.742-07:00 [Info] ClustMgr:handleCleanupPartition
Message: MsgCleanupPartition
Type: 67
Index defn Id: 3581653491356785672
Index inst Id: 5114813154892359787
Index partition Id: 0
Index replica Id: 0
Update Status Only: true
2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 3581653491356785672 instance id 5114813154892359787 real instance id 0 partitions [0]
2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 3581653491356785672 instance 5114813154892359787 partitions [0]
2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 3581653491356785672 instance id 5114813154892359787
2021-06-30T21:31:31.742-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance. Delete index 3581653491356785672
2021-06-30T21:40:35.986-07:00 [Info] LSS /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex(shard5) : LSSCtx.cleanup: Done cleanup
2021-06-30T21:40:35.986-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4bea8e0 IndexInstId 7889820684348130190 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex/recovery fatal: Corrupt lss log with tailOffset=1978368 >segmentOffset=1978367
2021-06-30T21:40:35.986-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 7889820684348130190 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-06-30T21:40:35.986-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
InstId: 7889820684348130190
Defn: DefnId: 4795002632039732232 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0
SecExprs: ([])
Desc: []
PartitionScheme: SINGLE
HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false
State: INDEX_STATE_ACTIVE
RState: RebalActive
Stream: MAINT_STREAM
saptarshisen@Saptarshis-MacBook-Pro data % ls -li n_2/data/@2i/default_\#primary_7889820684348130190_0.index/mainIndex/recovery
total 3880
15447438 -rwxr-xr-x 1 saptarshisen staff 8192 Jun 30 21:36 header.data
15447439 -rwxr-xr-x 1 saptarshisen staff 1048576 Jun 30 21:36 log.00000000000000.data
15447592 -rwxr-xr-x 1 saptarshisen staff 929792 Jun 30 21:36 log.00000000000001.data
saptarshisen@Saptarshis-MacBook-Pro data % truncate -s 929791 n_2/data/@2i/default_\#primary_7889820684348130190_0.index/mainIndex/recovery/log.00000000000001.data
Version: 0
ReplicaId: 0
PartitionContainer: &{map[0:{0 0 [:9117]}] 64 1 64 SINGLE 0} partnDefn {0 0 [:9117]}
2021-06-30T21:40:35.990-07:00 [Info] Indexer::initFromPersistedState Starting cleanup for PartitionId: 0 Endpoints: [:9117]
2021-06-30T21:40:35.990-07:00 [Info] Indexer::forceCleanupIndexPartition 7889820684348130190 0 mark metadata as deleted
Update Status Only: true
2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 4795002632039732232 instance id 7889820684348130190 real instance id 0 partitions [0]
2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 4795002632039732232 instance 7889820684348130190 partitions [0]
2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : index defnId 4795002632039732232 instance id 7889820684348130190
2021-06-30T21:40:35.990-07:00 [Info] LifecycleMgr.DeleteIndexInstance() : there is only a single instance. Delete index 4795002632039732232
2021-06-30T21:40:35.991-07:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CLEANUP_PARTITION elapsed 1.117917ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 error
2021-06-30T21:40:35.991-07:00 [Info] Indexer::forceCleanupIndexPartition Cleaning up data files for 7889820684348130190 0
Destory instances matching prefix /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index in /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i ...
2021-06-30T21:40:35.991-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : destroying instance /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex ...
2021-06-30T21:40:35.995-07:00 [Info] ServiceMgr::updateNodeList Updated Node List [46e686fe7ad138200a602836baf14164 e97b221f985c590c330d6a674ef0686d ef751350af236e270f66e34211da4441 0c3eec0fe078e88f1267f301fc92842b]
2021-06-30T21:40:35.995-07:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 1] [46e686fe7ad138200a602836baf14164 e97b221f985c590c330d6a674ef0686d ef751350af236e270f66e34211da4441 0c3eec0fe078e88f1267f301fc92842b] true []}
2021-06-30T21:40:35.995-07:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 1] []}
2021-06-30T21:40:35.998-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : removed plasmaId 1 for instance /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex ...
2021-06-30T21:40:35.999-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : metadata saved successfully
2021-06-30T21:40:35.999-07:00 [Info] Shard /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/shards/shard5(5) : instance /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index/mainIndex sucessfully destroyed
All instances matching prefix /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_2/data/@2i/default_#primary_7889820684348130190_0.index destroyed
2021-06-30T21:40:35.999-07:00 [Info] Indexer::forceCleanupIndexPartition 7889820684348130190 0 Cleanup partition in-memory data structure
2021-06-30T21:40:35.999-07:00 [Info] Indexer::forceCleanupIndexPartition 7889820684348130190 0 actually delete metadata
2021-06-30T21:40:35.999-07:00 [Info] ClustMgr:handleCleanupPartition
Message: MsgCleanupPartition
Type: 67
Index defn Id: 4795002632039732232
Index inst Id: 7889820684348130190
Index partition Id: 0
Index replica Id: 0
Update Status Only: false
2021-06-30T21:40:35.999-07:00 [Info] LifecycleMgr.DeleteOrPruneIndexInstance() : index defnId 4795002632039732232 instance id 7889820684348130190 real instance id 0 partitions [0]
2021-06-30T21:40:35.999-07:00 [Info] LifecycleMgr.PruneIndexPartition() : index defnId 4795002632039732232 instance 7889820684348130190 partitions [0]
Updated Patch:
2021-07-01T12:44:40.449-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x4beaa80 IndexInstId 18417244362498523011 fatal error occured: Unable to initialize /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_18417244362498523011_0.index/mainIndex, err = /Users/saptarshisen/ws/couchbase_svr/ns_server/data/n_0/data/@2i/default_#primary_18417244362498523011_0.index/mainIndex fatal: Corrupt lss log with end segmentOffset=83701759 < superblock tailOffset=83701760
-rwxr-xr-x 1 saptarshisen staff 1048576 Jul 1 12:32 log.00000000000078.data
drwxr-xr-x 5 saptarshisen staff 160 Jul 1 12:32 recovery
-rwxr-xr-x 1 saptarshisen staff 864256 Jul 1 12:32 log.00000000000079.data
-rwxr-xr-x 1 saptarshisen staff 8192 Jul 1 12:32 header.data
saptarshisen@Saptarshis-MacBook-Pro mainIndex % truncate -s 864255 log.00000000000079.data
Unit Tests:
saptarshisen@Saptarshis-MacBook-Pro plasma % go test -v -run TestLogMissingAndTruncatedSegment
Initializing write barrier = 8000
=== RUN TestLogMissingAndTruncatedSegments
(expected) error reading log file :test.TestLogMissingAndTruncatedSegments fatal: Corrupt lss log with start segmentOffset=1048576 >superblock headOffset=0
(expected) error reading log file :test.TestLogMissingAndTruncatedSegments fatal: Corrupt lss log with end segmentOffset=19922944 CREATE PRIMARY INDEX ON `default` using GSI;
{
"requestID": "c4c77700-46d6-4064-874d-26695db88214",
"signature": null,
"results": [
],
"status": "success",
"metrics": {
"elapsedTime": "8.573391324s",
"executionTime": "8.573337606s",
"resultCount": 0,
"resultSize": 0,
"serviceLoad": 2
}
}