Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59579

[System Test] Fatal errors seen - Mismatch in alternateId mapping

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      These fatal errors are seen during the system test run -

      On 176

      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2023-11-08T09:03:51.164-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in alternateId mapping. Expected alternateId mapping: 9163993064304961421-1-0 for shard: 14214665513905210707. New mapping is created with alternateId: 9163993064304961421-2-0, partn: 1, instId: 0
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2023-11-08T09:03:51.164-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in alternateId mapping. Expected alternateId mapping: 9163993064304961421-1-1 for shard: 13072267222003028533. New mapping is created with alternateId: 9163993064304961421-2-1, partn: 1, instId: 0
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2023-11-08T09:03:53.276-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in alternateId mapping. Expected alternateId mapping: 9163993064304961421-1-0 for shard: 14214665513905210707. New mapping is created with alternateId: 9163993064304961421-2-0, partn: 5, instId: 0
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2023-11-08T09:03:53.276-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in alternateId mapping. Expected alternateId mapping: 9163993064304961421-1-1 for shard: 13072267222003028533. New mapping is created with alternateId: 9163993064304961421-2-1, partn: 5, instId: 0
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2023-11-08T09:04:50.199-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in shard mapping. Expected shard mapping: 17813642152398285475 for alternateId: 3964352297181240124-2-0. New mapping is created with shardId: 15291845628819952590, partn: 4, instId: 0
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2023-11-08T09:04:50.199-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in shard mapping. Expected shard mapping: 15235249916196832371 for alternateId: 3964352297181240124-2-1. New mapping is created with shardId: 4643909379619616715, partn: 4, instId: 0

      On 109

      opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-11-08T09:06:18.594-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in alternateId mapping. Expected alternateId mapping: 1420597029539389819-1-0 for shard: 6235732958143713416. New mapping is created with alternateId: 1420597029539389819-2-0, partn: 4, instId: 0
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2023-11-08T09:06:18.594-08:00 [Fatal] LifecycleMgr::assertShardMappings Mismatch in alternateId mapping. Expected alternateId mapping: 1420597029539389819-1-1 for shard: 6090895495012142041. New mapping is created with alternateId: 1420597029539389819-2-1, partn: 4, instId: 0

      On 108, we see errors like these -

      /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz:2023-11-08T08:50:01.444-08:00 [Error] plasmaSlice:NewplasmaSlice Id 0x161bc20 IndexInstId 17230545408771517438 fatal error occured: Unable to initialize /data/@2i/bucket2_idx13_U4UBxsD_17230545408771517438_4.index/mainIndex, err = fatal: Fail to find shard for shared instance /data/@2i/bucket2_idx13_U4UBxsD_17230545408771517438_4.index/mainIndex due to instance (shared) not present on disk and no failed/corrupted shards
      		 
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz:2023-11-08T08:50:01.444-08:00 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 17230545408771517438 PartitionId 4 isNew falsefatal error occured: Storage path not found for recovery

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.106.171.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.106.176.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1699463374/collectinfo-2023-11-08T171830-ns_1%40172.23.97.67.zip

      Attachments

        For Gerrit Dashboard: MB-59579
        # Subject Branch Project Status CR V
        200563,3 MB-59579 Disable assertShardMappings at lifecycle manager unstable indexing Status: MERGED +2 +1

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty