Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29237

System Test : Panic in indexer while building index

    XMLWordPrintable

Details

    Description

      Build : 5.5.0-2473

      The integration longevity test now has the changes to include partitioned indexes and the queries have been updated. One of the step is to drop a few indexes and recreate them. While creating one index (non-partitioned), build index got stuck and ultimately indexer crashed. There were panic messages in atleast 2 indexer nodes.

      2018-04-12T02:15:00.714-07:00 [Info] DATP[->dataport ":9103"] DATP -> Indexer 88.762809% blocked
      		 
      panic: default/default_rating/Mainstore : fatal: Log starts at offset 353052900, trying to read 350285654
      		 
       
      		 
      goroutine 3869 [running]:
      		 
      panic(0xdbf4e0, 0xc469989860)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:500 +0x1a1 fp=0xc5b8ec51e8 sp=0xc5b8ec5158
      		 
      github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc4418cc380, 0xc4362bef80, 0x3f)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1701 +0x289 fp=0xc5b8ec52c8 sp=0xc5b8ec51e8
      		 
      github.com/couchbase/plasma.(*pageWalker).readPageSegment(0xc494254380)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_walker.go:161 +0x344 fp=0xc5b8ec5370 sp=0xc5b8ec52c8
      		 
      github.com/couchbase/plasma.(*pageWalker).Next(0xc494254380)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_walker.go:147 +0x61 fp=0xc5b8ec5390 sp=0xc5b8ec5370
      		 
      github.com/couchbase/plasma.newPgOpIterator(0x7f3855daf380, 0x102e4e8, 0x0, 0x7f377ba0f910, 0x17ee4a0, 0xc4362be7c0, 0xc44137ba40, 0xc469989800, 0x416f1e, 0xc499aacf00)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/iterator.go:432 +0x268 fp=0xc5b8ec5590 sp=0xc5b8ec5390
      		 
      github.com/couchbase/plasma.newPgOpIterator(0x7f36f0de4da0, 0x102e4e8, 0x0, 0x7f377ba0f910, 0x17ee4a0, 0xc4362be7c0, 0xc44137ba40, 0xc5b8ec57f0, 0x7f389ac7b9bf, 0x12)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/iterator.go:448 +0xb18 fp=0xc5b8ec5790 sp=0xc5b8ec5590
      		 
      github.com/couchbase/plasma.(*page).collectItems(0xc4362be780, 0x7f36f0de4da0, 0x0, 0x7f377ba0f910, 0xc500533601, 0x0, 0x0, 0x436459, 0xc42bb9bc58, 0x200000003, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page.go:755 +0xb7 fp=0xc5b8ec5838 sp=0xc5b8ec5790
      		 
      github.com/couchbase/plasma.(*page).Compact(0xc4362be780, 0xc8)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page.go:668 +0x6c fp=0xc5b8ec5900 sp=0xc5b8ec5838
      		 
      github.com/couchbase/plasma.(*Plasma).trySMOs2(0xc4418cc380, 0x7f382405d0a0, 0x180a4c0, 0xc4362be780, 0xc44137ba40, 0xc5b8ec5a01, 0x190, 0x19, 0xc8, 0x4, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1451 +0x83 fp=0xc5b8ec5a50 sp=0xc5b8ec5900
      		 
      github.com/couchbase/plasma.(*Plasma).trySMOs(0xc4418cc380, 0x7f382405d0a0, 0x180a4c0, 0xc4362be780, 0xc44137ba40, 0xc4362be701, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1442 +0x96 fp=0xc5b8ec5ab8 sp=0xc5b8ec5a50
      		 
      github.com/couchbase/plasma.(*Writer).Insert(0xc43f2684c0, 0xc46de2e000, 0x21, 0x3702)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1580 +0xf5 fp=0xc5b8ec5b38 sp=0xc5b8ec5ab8
      		 
      github.com/couchbase/plasma.(*Writer).DeleteKV(0xc43f2684c0, 0xc4468fe000, 0x21, 0x3702, 0xf, 0xf)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mvcc.go:319 +0x10f fp=0xc5b8ec5bb8 sp=0xc5b8ec5b38
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).deleteSecIndex(0xc44655d500, 0xc5816316e0, 0x12, 0x12, 0x0, 0x0, 0x0, 0x7, 0x0, 0x5acf2300)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:782 +0x4a1 fp=0xc5b8ec5c80 sp=0xc5b8ec5bb8
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).delete(0xc44655d500, 0xc5816316e0, 0x12, 0x12, 0x7, 0x1)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:724 +0xf8 fp=0xc5b8ec5ce8 sp=0xc5b8ec5c80
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc44655d500, 0x7)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:421 +0x396 fp=0xc5b8ec5f40 sp=0xc5b8ec5ce8
      		 
      runtime.goexit()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc5b8ec5f48 sp=0xc5b8ec5f40
      		 
      created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:175 +0xdf9

      Logs attached

      Attachments

        Issue Links

          is triggered by

          Bug - A problem which impairs or prevents the functions of the product. MB-28868 plasma system test: fatal: Log starts at offset 1374415487, trying to read 1367854282

          • Test Blocker - QE-related tests can't proceed
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sarath Sarath Lakshman
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty