Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30461

[System test] indexer crash with panic : fatal: Got empty read at offset 1110418737

    XMLWordPrintable

Details

    • Storage-Sprint-Dec-21-2018, Storage-Sprint-Jan-25-2019, Storage-Sprint-Feb-08-2019, Storage-Sprint-Mar-15-2019, Plasma-Sprint-May-30-2021

    Description

      Build : 5.5.0-2958 (RC4)
      Test : -test tests/integration/test_allFeatures_vulcan.yml -scope tests/integration/scope_Xattrs_Vulcan.yml
      Scale : 3
      Iteration : 6 (1d 20h)

      Seeing a panic caused at plasma layer. Indexer crashed because of this, but recovered subsequently after deleting the corrupt index.

      This is on node 172.23.104.69

      panic: default/default_result_ratings_claims_pindex/Backstore#6268538637229155700:4 : fatal: Got empty read at offset 1110418737
      		 
       
      		 
      goroutine 1053 [running]:
      		 
      panic(0xe22440, 0xc52537e210)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1 fp=0xc524618a50 sp=0xc5246189c0
      		 
      github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc423d68800, 0xc455b341e0, 0x23)
      		 
              goproj/src/github.com/couchbase/plasma/plasma.go:1810 +0x2f5 fp=0xc524618f10 sp=0xc524618a50
      		 
      github.com/couchbase/plasma.(*Plasma).fetchPageSegmentFromLSS(0xc423d68800, 0x422fa531, 0xc429f99d60, 0xc4cbcca370, 0xc42c444060, 0x417818, 0x40, 0xf1da40, 0xc42e7b6401, 0xc4ff7e2e80)
      		 
              goproj/src/github.com/couchbase/plasma/plasma.go:1762 +0x736 fp=0xc524618fe8 sp=0xc524618f10
      		 
      github.com/couchbase/plasma.(*Plasma).(github.com/couchbase/plasma.fetchPageSegmentFromLSS)-fm(0x422fa531, 0xc429f99d60, 0xc4cbcca370, 0xc42c444060, 0x40927a, 0xc42e7b6400, 0xc4390f9108, 0xc400000000, 0xdca03ee0)
      		 
              goproj/src/github.com/couchbase/plasma/plasma.go:1182 +0x56 fp=0xc524619048 sp=0xc524618fe8
      		 
      github.com/couchbase/plasma.(*pageWalker).readPageSegment(0xc5246191f0)
      		 
              goproj/src/github.com/couchbase/plasma/page_walker.go:159 +0x99 fp=0xc524619120 sp=0xc524619048
      		 
      github.com/couchbase/plasma.(*pageWalker).Next(0xc5246191f0)
      		 
              goproj/src/github.com/couchbase/plasma/page_walker.go:148 +0x61 fp=0xc524619140 sp=0xc524619120
      		 
      github.com/couchbase/plasma.(*page).computeItemsCountMVCC(0xc4ff7e2e40, 0xc429f99d60, 0x7f508df7c440, 0x7f508df7c468, 0x0, 0x0)
      		 
              goproj/src/github.com/couchbase/plasma/page.go:1540 +0x28f fp=0xc5246192b0 sp=0xc524619140
      		 
      github.com/couchbase/plasma.(*page).NeedPurging(0xc4ff7e2e40, 0x401c000000000000, 0xc429f99d00)
      		 
              goproj/src/github.com/couchbase/plasma/page.go:597 +0x56 fp=0xc5246192f0 sp=0xc5246192b0
      		 
      github.com/couchbase/plasma.(*Plasma).PurgePages.func1(0x7f513dfabb70, 0x7f50e2683c00)
      		 
              goproj/src/github.com/couchbase/plasma/mvcc_purger.go:47 +0x1bf fp=0xc524619768 sp=0xc5246192f0
      		 
      github.com/couchbase/plasma.(*PageRingVisitor).Exec(0xc4291dd120, 0xc524619818)
      		 
              goproj/src/github.com/couchbase/plasma/page_visitor.go:154 +0x30a fp=0xc5246197e8 sp=0xc524619768
      		 
      github.com/couchbase/plasma.(*Plasma).PurgePages(0xc423d68800, 0x401c000000000000)
      		 
              goproj/src/github.com/couchbase/plasma/mvcc_purger.go:65 +0x104 fp=0xc524619850 sp=0xc5246197e8
      		 
      github.com/couchbase/plasma.(*Plasma).mvccPurgerDaemon(0xc423d68800)
      		 
              goproj/src/github.com/couchbase/plasma/mvcc_purger.go:83 +0x349 fp=0xc524619f88 sp=0xc524619850
      		 
      runtime.goexit()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc524619f90 sp=0xc524619f88
      		 
      created by github.com/couchbase/plasma.New
      		 
              goproj/src/github.com/couchbase/plasma/plasma.go:613 +0x12cf

       

      The test had just finished adding an indexer node to the cluster before this crash happened (2018-07-11T21:42:01). Excerpt from the test logs :

      [2018-07-11T21:30:43-07:00, sequoiatools/couchbase-cli:ac659a] rebalance -c 172.23.108.103:8091 --server-remove 172.23.96.148 -u Administrator -p password
      		 
      [2018-07-11T21:35:56-07:00, sequoiatools/cmd:258cb8] 60
      		 
      [2018-07-11T21:37:15-07:00, sequoiatools/cmd:e0cccd] 300
      		 
      [2018-07-11T21:53:37-07:00, sequoiatools/couchbase-cli:705869] server-add -c 172.23.108.103:8091 --server-add 172.23.96.148:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services index
      		 
      [2018-07-11T21:54:07-07:00, sequoiatools/couchbase-cli:f1e142] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.238 -u Administrator -p password
      		 
      [2018-07-11T22:00:19-07:00, sequoiatools/cmd:f4a979] 60

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jliang John Liang
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty