Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47752

Storage corruption in log messages after a panic

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • Morpheus
    • None
    • storage-engine
    • Untriaged
    • 1
    • No

    Description

      When running the indexer test set14_rebalance_test.go, the following panic was seen and is reported in https://issues.couchbase.com/browse/MB-47749.

      goroutine 166 [running]:
      runtime.throw(0x131592e, 0x1e)
      {{ /home/buildbot/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/panic.go:1117 +0x72 fp=0xc006507ba0 sp=0xc006507b70 pc=0x43fdd2}}
      sync.throw(0x131592e, 0x1e)
      {{ /home/buildbot/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/panic.go:1103 +0x35 fp=0xc006507bc0 sp=0xc006507ba0 pc=0x474375}}
      sync.(*Mutex).unlockSlow(0xc004e96610, 0xffffffff)
      {{ /home/buildbot/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/sync/mutex.go:196 +0xd8 fp=0xc006507be8 sp=0xc006507bc0 pc=0x491e18}}
      sync.(*Mutex).Unlock(...)
      {{ /home/buildbot/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/sync/mutex.go:190}}
      github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleIndexMergeSnapshot(0xc004e96580, 0x14a4a80, 0xc00cdd0ff0)
      {{ /opt/build/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1720 +0x699 fp=0xc006507f20 sp=0xc006507be8 pc=0xfb67f9}}
      github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleSupvervisorCommands(0xc004e96580, 0x14a4a80, 0xc00cdd0ff0)
      {{ /opt/build/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:224 +0x1b2 fp=0xc006507f58 sp=0xc006507f20 pc=0xfab8f2}}
      github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0xc004e96580)
      {{ /opt/build/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:182 +0x48 fp=0xc006507fd8 sp=0xc006507f58 pc=0xfab628}}
      runtime.goexit()
      {{ /home/buildbot/.cbdepscache/exploded/x86_64/go-1.16.5/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc006507fe0 sp=0xc006507fd8 pc=0x4792e1}}
      created by }}{{github.com/couchbase/indexing/secondary/indexer.NewStorageManager
      {{ /opt/build/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2e5}} 

       

      After the panic, the following error messages appear in the log

      17213:2021-08-04T13:20:49.204+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0x102eaa0 IndexInstId 7292211632063708996 fatal error occured: Unable to initialize /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_guid_balance_7292211632063708996_5.index/mainIndex, err = fatal: Fail to find shard for dedicated instance /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_guid_balance_7292211632063708996_5.index/mainIndex due to instance not present on disk
      17214:2021-08-04T13:20:49.204+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 7292211632063708996 PartitionId 5 fatal error occured: Storage corrupted and unrecoverable
      17231:2021-08-04T13:20:49.205+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0x102eaa0 IndexInstId 7292211632063708996 fatal error occured: Unable to initialize /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_guid_balance_7292211632063708996_6.index/mainIndex, err = fatal: Fail to find shard for dedicated instance /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_guid_balance_7292211632063708996_6.index/mainIndex due to instance not present on disk
      17232:2021-08-04T13:20:49.205+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 7292211632063708996 PartitionId 6 fatal error occured: Storage corrupted and unrecoverable
      17551:2021-08-04T13:20:49.534+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0x102eaa0 IndexInstId 8114273593663889297 fatal error occured: Unable to initialize /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_docid_isActive_8114273593663889297_6.index/mainIndex, err = fatal: Fail to find shard for dedicated instance /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_docid_isActive_8114273593663889297_6.index/mainIndex due to instance not present on disk
      17552:2021-08-04T13:20:49.534+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 8114273593663889297 PartitionId 6 fatal error occured: Storage corrupted and unrecoverable
      17569:2021-08-04T13:20:49.535+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0x102eaa0 IndexInstId 8114273593663889297 fatal error occured: Unable to initialize /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_docid_isActive_8114273593663889297_7.index/mainIndex, err = fatal: Fail to find shard for dedicated instance /opt/build/ns_server/data/n_3/data/@2i/default_set14_idx_7PARTITIONS_docid_isActive_8114273593663889297_7.index/mainIndex due to instance not present on disk
      17570:2021-08-04T13:20:49.535+05:30 [Error] plasmaSlice:NewplasmaSlice Id 0 IndexInstId 8114273593663889297 PartitionId 7 fatal error occured: Storage corrupted and unrecoverable
       
      Need to investigate if this is a real plasma issue.

       

      Attachments

        For Gerrit Dashboard: MB-47752
        # Subject Branch Project Status CR V
        170730,2 MB-47752: Provide IsErrorRecoveryInstPathNotFound check for fatal errors unstable plasma Status: NEW 0 -1
        170731,3 MB-47752: Add error errStoragePathNotFound during plasma recovery unstable indexing Status: NEW 0 0

        Activity

          People

            saptarshi.sen Saptarshi Sen
            srinath.duvuru Srinath Duvuru
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:

              PagerDuty