Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44556

panic in indexer : [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • Plasma-Sprint-May-30-2021

    Description

      Seeing this panic in Scan Latency Tests for 5 indexes using PLASMA 

      2021-02-23T22:23:46.867-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(>>224->>3159->>3104->>3194-)</ud> docid = <ud>(gsi-000031628254)</ud> Index myindex5, Bucket bucket-1, IndexInstId 9022482831798013220, PartitionId 02021-02-23T22:23:46.867-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(>>224->>3159->>3104->>3194-)</ud> docid = <ud>(gsi-000031628254)</ud> Index myindex5, Bucket bucket-1, IndexInstId 9022482831798013220, PartitionId 02021-02-23T22:23:46.886-08:00 [Fatal] goroutine 2163 [running]:github.com/couchbase/indexing/secondary/logging.(*destination).StackTraceAll(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:175github.com/couchbase/indexing/secondary/logging.StackTraceAll(0x20f0630, 0x2110001) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:317 +0x6dgithub.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1(0xc0ca3a7ed0, 0xc00591cc00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:596 +0x424panic(0x104e3a0, 0xc05457b580) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2github.com/couchbase/plasma.(*item).checkInvalidHeader(0x7febd4d8215a) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:242 +0x11fgithub.com/couchbase/plasma.(*item).Key(0x7febd4d8215a, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:98 +0x3cgithub.com/couchbase/plasma.cmpItem(0x7ff07bd37480, 0x7febd4d8215a, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:182 +0xa8github.com/couchbase/plasma.(*page).SatisfyHighRange(0xc0a85803c0, 0x7ff07bd37480, 0x12e8801) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page.go:615 +0x45github.com/couchbase/plasma.(*Plasma).fetchPage(0xc00591d180, 0x7ff07bd37480, 0xc042d8ab00, 0xc0c95bf070, 0x10, 0xd06001, 0xc005a18c60, 0x7ff07bd37480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1271 +0x30fgithub.com/couchbase/plasma.(*Writer).Lookup(0xc0060c77a0, 0x7ff07bd37480, 0x10, 0x10, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/writer.go:124 +0x63github.com/couchbase/plasma.(*Writer).LookupKV(0xc0060c77a0, 0xc0c95bf070, 0x10, 0x10, 0x43dcdc, 0xc00611ef00, 0x300000002, 0xc00611ef00, 0xc0ca3a7978) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mvcc.go:395 +0x10cgithub.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).deleteSecIndex(0xc00591cc00, 0xc0c95bf070, 0x10, 0x10, 0xc0ad7f0990, 0x27, 0x30, 0xb, 0x0, 0x10000c0060fd000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1141 +0x13bgithub.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecIndex(0xc00591cc00, 0xc0ad7f0990, 0x27, 0x30, 0xc0c95bf070, 0x10, 0x10, 0xb, 0x4c2400, 0xc06eea17c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:775 +0xb18github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0xc00591cc00, 0xc0ad7f0990, 0x27, 0x30, 0xc0c95bf070, 0x10, 0x10, 0xb, 0xc005032700, 0xc06eea17c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:733 +0x164github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc00591cc00, 0xb) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:610 +0x58acreated by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2969 +0x521
      

       

      The run : http://perf.jenkins.couchbase.com/job/secondary-plasma/6184/consoleFull 

      Logs :  

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-secondary-plasma-6184/172.23.97.10.zip 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jliang John Liang
              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty