Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57391

Panic detected in indexer - plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown
    • Plasma-Sprint-Aug-15-2023

    Description

      The stack trace is pretty long so here's an excerpt -

      2023-06-13T16:20:33.644+00:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key  Index idx10_zyD1ucp, Bucket default5, IndexInstId 9257112166260831750, PartitionId 7
      2023-06-13T16:20:33.699+00:00 [Fatal] goroutine 240468 [running]:
      github.com/couchbase/indexing/secondary/logging.(*destination).StackTraceAll(...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:175
      github.com/couchbase/indexing/secondary/logging.StackTraceAll()
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:317 +0x3f
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1()
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:666 +0x30f
      panic({0x1032100, 0xc0cbdc3e20})
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/panic.go:838 +0x207
      github.com/couchbase/plasma.(*allocCtx).Reset(0xc02f916e00?)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_alloc.go:150 +0x146
      github.com/couchbase/plasma.(*Plasma).freeWCtx(0xc02f916e00, 0x1cea, 0xc0382fdcb0)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/wctx.go:578 +0x74
      github.com/couchbase/plasma.(*Writer).freeCtx(0xc037c33500)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/writer.go:218 +0x47
      github.com/couchbase/plasma.(*Writer).End(0xc037c33500)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/writer.go:77 +0x92
      panic({0x1069100, 0xc0cbdc2f20})
      	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.18.7/go/src/runtime/panic.go:844 +0x258
      github.com/couchbase/plasma.(*Shard).raisePanic(...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1051
      github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc02f916e00, {0xc0f3add600, 0x157})
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1645 +0x316
      github.com/couchbase/plasma.(*Plasma).trySMOs2(0xc02f916e00, {0xc0382fdcb0?}, {0x1450fd8, 0xc068511680}, 0xc0382fdcb0, 0x1, 0x200?, 0x600?, 0x702?, 0x4)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1400 +0x578
      github.com/couchbase/plasma.(*Plasma).trySMOs(...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1352
      github.com/couchbase/plasma.(*Writer).Insert(0xc037c33500, 0x0?)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/writer.go:101 +0x145
      github.com/couchbase/plasma.(*Writer).InsertKV(0xc037c33500, {0xc028251800, 0x24, 0x702}, {0x0, 0x0, 0x0})
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mvcc.go:473 +0x225
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecArrayIndex(0xc01ae72c00, {0xc0968d2400, 0x3d9, 0x400}, {0xc0489e60b0, 0xf, 0x10}, 0x1, 0x0, 0xc0f8d70100)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1090 +0x2273
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0xc01ae72c00, {0xc0968d2400?, 0x0?, 0x2?}, {0xc0489e60b0?, 0x7fa496e52a24?, 0xc0248a1f38?}, 0xcdf767?, 0xb0?, 0xc0f8d70100)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:805 +0xa6
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc01ae72c00, 0x1)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:680 +0x425
      created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:3213 +0x3cd
      

      https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-001.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-002.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-003.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-d-node-006.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-qi-node-004.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-qi-node-005.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/Systest_Jun12_GSI_2_3/collectinfo-2023-06-13T162154-ns_1%40svc-qi-node-007.yu7y8akx02-zpqw.sandbox.nonprod-project-avengers.com.zip

      This was found on 7.1.4 during the system test development phase while I wasn't paying attention to the version (I'm now using 7.2.0). The fix version can be updated after RCA Amit Kulkarni

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty