Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-54510

Swap Rebalance of failed over node failed

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • Elixir
    • Elixir
    • secondary-index
    • CBS 7.5.0 build 3227
    • Untriaged
    • 0
    • Unknown

    Description

      Swap Rebalance of of failed over node failed with below error - 

       

      Rebalance exited with reason {service_rebalance_failed,index, {worker_died, {'EXIT',<0.17941.1>, {rebalance_failed, {service_error, <<"Build index fails. Index #primary fails to build for reason: KeyspaceId: test_bucket_1:_system:_query not found when computing the restartTs. All available keyspaces: [].">>}}}}}. Rebalance Operation Id = ab42eff360bfae19628d70a437168ad3 

       

       

      Steps to validate:

      1. Create 3 node cluster with Kv:N1QL-Index-Index in three server groups SG1:SG2:SG3
      2. Create 2 buckets and load data and create few indexes for each.
      3. Failover index node in SG2 server group and add a new index node in SG2 server group.
      4. Rebalance the node.

      Attachments

        1. node1-cb710-debian10.zip
          4.36 MB
        2. node2-cb710-debian10.zip
          3.29 MB
        3. node3-cb710-debian10.zip
          3.61 MB
        4. node4-cb710-debian10.zip
          2.54 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Duplicate of MB-54411.

          Error while recovering #primary index

          2022-11-10T04:37:34.063-05:00 [Info] Indexer::initPartnInstance Initialized Partition:
                   Index: 376372267988293948 Partition: PartitionId: 0 Endpoints: [:9105]
          2022-11-10T04:37:34.065-05:00 [Error] plasmaSlice:NewplasmaSlice Id 0x13ee060 IndexInstId 376372267988293948 fatal error occured: Unable to initialize /opt/couchbase/var/lib/couchbase/data/@2i/test_bucket_1_#primary_376372267988293948_0.index/mainIndex, err = fatal: Fail to find shard for shared instance /opt/couchbase/var/lib/couchbase/data/@2i/test_bucket_1_#primary_376372267988293948_0.index/mainIndex due to instance not present on disk
          2022-11-10T04:37:34.066-05:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 376372267988293948 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable

          In MB-53890, we have improved logging for such cases

          varun.velamuri Varun Velamuri added a comment - Duplicate of MB-54411. Error while recovering #primary index 2022-11-10T04:37:34.063-05:00 [Info] Indexer::initPartnInstance Initialized Partition:          Index: 376372267988293948 Partition: PartitionId: 0 Endpoints: [:9105] 2022-11-10T04:37:34.065-05:00 [Error] plasmaSlice:NewplasmaSlice Id 0x13ee060 IndexInstId 376372267988293948 fatal error occured: Unable to initialize /opt/couchbase/var/lib/couchbase/data/@2i/test_bucket_1_#primary_376372267988293948_0.index/mainIndex, err = fatal: Fail to find shard for shared instance /opt/couchbase/var/lib/couchbase/data/@2i/test_bucket_1_#primary_376372267988293948_0.index/mainIndex due to instance not present on disk 2022-11-10T04:37:34.066-05:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 376372267988293948 PartitionId 0 fatal error occured: Storage corrupted and unrecoverable In MB-53890, we have improved logging for such cases

          Looking further into the logs, the issue is because of MB-53754, which is fixed in https://review.couchbase.org/c/indexing/+/182420. The issue is still a duplicate though.

          varun.velamuri Varun Velamuri added a comment - Looking further into the logs, the issue is because of MB-53754, which is fixed in https://review.couchbase.org/c/indexing/+/182420. The issue is still a duplicate though.

          People

            hemant.rajput Hemant Rajput
            hemant.rajput Hemant Rajput
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty