Details
-
Bug
-
Resolution: User Error
-
Critical
-
None
-
6.6.3
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
Build - 6.6.3-9791
Longevity Test - -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Day - 3
Scale - 3
On 172.23.96.243
- Indexer started at 2021-07-14T15:47:05
2021-07-14T15:47:05.531-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/data/couchbase/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=b23fd07bfc020ed33c5e4d9808eb7b36 -isEnterprise=true]
- Errors during recovery
Plasma: Fail to recover /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex using recovery log due to error default/default_result/Backstore#712400206795173573:0 : fatal: found missing page. lastPg minItem (item key:<ud>(617B63BE-153_752676)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_75293)</ud> val:<ud>((nil))</ud> sn:1 insert: false). pg minItem (item key:<ud>(617B63BE-153_75387)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_7548)</ud> val:<ud>((nil))</ud> sn:1 insert: false). Perform full data replay.
2021-07-14T15:47:23.486-07:00 [Info] LSS /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex(shard2) : recoverFromDataReplay: Begin recovering from data log, headOffset [2418839925] tailOffset [2463924224] replayOffset [0]
2021-07-14T15:47:23.486-07:00 [Info] default/default_result/Backstore#712400206795173573:0 Plasma: Full data replay fails on /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex. Error: Log starts at offset 2418839925, trying to read 0
2021-07-14T15:47:23.843-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0x1038dc0 IndexInstId 712400206795173573 fatal error occured: Unable to initialize /data/couchbase/@2i/default_default_result_712400206795173573_0.index/docIndex, err = default/default_result/Backstore#712400206795173573:0 : fatal: found missing page. lastPg minItem (item key:<ud>(617B63BE-153_752676)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_75293)</ud> val:<ud>((nil))</ud> sn:1 insert: false). pg minItem (item key:<ud>(617B63BE-153_75387)</ud> val:<ud>((nil))</ud> sn:1 insert: false) maxItem (item key:<ud>(617B63BE-153_7548)</ud> val:<ud>((nil))</ud> sn:1 insert: false)
2021-07-14T15:47:23.843-07:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 712400206795173573 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable
2021-07-14T15:47:23.843-07:00 [Error] Indexer:: initPartnInstance storage corruption for indexInst
InstId: 712400206795173573
Defn: DefnId: 17676812743065794975 Name: default_result Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: false NumReplica: 0 InstVersion: 0
SecExprs: <ud>([`result`])</ud>
Desc: [false]
PartitionScheme: SINGLE
HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false
State: INDEX_STATE_ACTIVE
RState: RebalActive
Stream: MAINT_STREAM
Version: 0
ReplicaId: 0
PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} partnDefn {0 0 [:9105]}