Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60917

[System Test] Panic in indexer - Error observed when processing Statistics

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      Panic seen on 109 -

      2024-02-22T06:35:48.546-08:00 [Info] Shard /data/@2i/shards/shard10523355994537715022(10523355994537715022)(6012689926613359440-0-0) : Shard.closeInstance: instance /data/@2i/bucket3_idx3_pOzo_10073953503448448880_5.index/mainIndex closed
      2024-02-22T06:35:48.547-08:00 [Info] bucket3/idx3_pOzo/Mainstore#10073953503448448880:5 Plasma: Plasma.Close: Done closing plasma instance
      2024-02-22T06:35:48.547-08:00 [Info] bucket3/idx3_pOzo/Backstore#10073953503448448880:5 Plasma: Plasma.Close: Begin closing plasma instance
      2024-02-22T06:35:48.548-08:00 [Fatal] PlasmaSlice::Statistics Error observed when processing Statistics on instId: 11118697127090909452, partnId: 0
      2024-02-22T06:35:48.548-08:00 [Info] bucket3/idx3_pOzo/Backstore#10073953503448448880:5 Plasma: Plasma.monitor: Received stop message
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11e4133]
       
      goroutine 8269396 [running]:
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics.func1()
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2883 +0xd4
      panic({0x17ccc00?, 0x2e35180?})
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.21.6/go/src/runtime/panic.go:914 +0x21f
      github.com/couchbase/nitro/skiplist.(*Skiplist).GetStats(...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/stats.go:139
      github.com/couchbase/plasma.(*Plasma).GetNumPages(...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1158
      github.com/couchbase/plasma.(*Plasma).getStats(_, _)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1171 +0xf3
      github.com/couchbase/plasma.(*Plasma).GetPreparedStats(...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1153
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).Statistics(0xc00abc4600, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2898 +0x11c
      github.com/couchbase/indexing/secondary/indexer.(*storageMgr).getIndexStorageStats(0xc0059a8340, 0x0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1843 +0xb9e
      github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats.func1()
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1596 +0x105
      created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleStats in goroutine 329
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1590 +0xb6
      

      On cursory glance, even though we have seen a similar panic before (https://issues.couchbase.com/browse/MB-60580 ), this does not look similar to MB-60580, but RCA can help determine if it is.

      cbcollect -

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1708612686/collectinfo-2024-02-22T144622-ns_1%40172.23.97.67.zip

      cc Varun Velamuri Ritam Sharma

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty