[System Test] fatal error occured: Storage corrupted and unrecoverable.

Build - 6.6.3-9791
Longevity Test - -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Day - 3
Scale - 3

On 172.23.96.243

• Indexer started at 2021-07-14T15:47:05

  2021-07-14T15:47:05.531-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/data/couchbase/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=b23fd07bfc020ed33c5e4d9808eb7b36 -isEnterprise=true]

• Errors during recovery

  Plasma: Fail to recover /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex using recovery log due to error default/default_result/Backstore#712400206795173573:0 : fatal: found missing page. lastPg minItem (item key:<ud>(617B63BE-153_752676)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_75293)</ud> val:<ud>((nil))</ud> sn:1 insert: false). pg minItem (item key:<ud>(617B63BE-153_75387)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_7548)</ud> val:<ud>((nil))</ud> sn:1 insert: false). Perform full data replay.

  2021-07-14T15:47:23.486-07:00 [Info] LSS /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex(shard2) : recoverFromDataReplay: Begin recovering from data log, headOffset [2418839925] tailOffset [2463924224] replayOffset [0]

  2021-07-14T15:47:23.486-07:00 [Info] default/default_result/Backstore#712400206795173573:0 Plasma: Full data replay fails on /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex. Error: Log starts at offset 2418839925, trying to read 0

  2021-07-14T15:47:23.843-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x1038dc0 IndexInstId 712400206795173573 fatal error occured: Unable to initialize /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex, err = default/default_result/Backstore#712400206795173573:0 : fatal: found missing page. lastPg minItem (item key:<ud>(617B63BE-153_752676)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_75293)</ud> val:<ud>((nil))</ud> sn:1 insert: false). pg minItem (item key:<ud>(617B63BE-153_75387)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_7548)</ud> val:<ud>((nil))</ud> sn:1 insert: false)

  2021-07-14T15:47:23.843-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 712400206795173573 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable

  2021-07-14T15:47:23.843-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst

  InstId: 712400206795173573

  Defn: DefnId: 17676812743065794975 Name: default_result Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: false NumReplica: 0 InstVersion: 0

  SecExprs: <ud>([`result`])</ud>

  Desc: [false]

  PartitionScheme: SINGLE

  HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false

  State: INDEX_STATE_ACTIVE

  RState: RebalActive

  Stream: MAINT_STREAM

  Version: 0

  ReplicaId: 0

  PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} partnDefn {0 0 [:9105]}