Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
Elixir
-
CBS 7.5.0 build 3227
-
Untriaged
-
0
-
Unknown
Description
Swap Rebalance of of failed over node failed with below error -
Rebalance exited with reason {service_rebalance_failed,index, {worker_died, {'EXIT',<0.17941.1>, {rebalance_failed, {service_error, <<"Build index fails. Index #primary fails to build for reason: KeyspaceId: test_bucket_1:_system:_query not found when computing the restartTs. All available keyspaces: [].">>}}}}}. Rebalance Operation Id = ab42eff360bfae19628d70a437168ad3 |
Steps to validate:
- Create 3 node cluster with Kv:N1QL-Index-Index in three server groups SG1:SG2:SG3
- Create 2 buckets and load data and create few indexes for each.
- Failover index node in SG2 server group and add a new index node in SG2 server group.
- Rebalance the node.
Attachments
Activity
Looking further into the logs, the issue is because of MB-53754, which is fixed in https://review.couchbase.org/c/indexing/+/182420. The issue is still a duplicate though.
Duplicate of MB-54411.
Error while recovering #primary index
2022-11-10T04:37:34.063-05:00 [Info] Indexer::initPartnInstance Initialized Partition:Index: 376372267988293948 Partition: PartitionId: 0 Endpoints: [:9105]2022-11-10T04:37:34.065-05:00 [Error] plasmaSlice:NewplasmaSlice Id 0x13ee060 IndexInstId 376372267988293948 fatal error occured: Unable to initialize /opt/couchbase/var/lib/couchbase/data/@2i/test_bucket_1_#primary_376372267988293948_0.index/mainIndex, err = fatal: Fail to find shard for shared instance /opt/couchbase/var/lib/couchbase/data/@2i/test_bucket_1_#primary_376372267988293948_0.index/mainIndex due to instance not present on disk2022-11-10T04:37:34.066-05:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 376372267988293948 PartitionId 0 fatal error occured: Storage corrupted and unrecoverableIn MB-53890, we have improved logging for such cases