Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44749

[System Test] : Plasma crash - plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-4603
      Test : Longevity (-test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml)
      Scale : 2
      Iteration : 1st

      On 172.23.96.252, seeing the following errors leading to indexer crash while rebalance is going on. In-memory compression is not turned on in this test.

      2021-03-04T10:06:09.702-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(^H^F_sync:unusedSeq:56555095^@^@^@)</ud> docid = <ud>(_sync:unusedSeq:56555095)</ud> Index sg_syncDocs_x1, Bucket default, IndexInstId 13320080993727085165, PartitionId 0
      		 
      ...
      		 
      ...
      		 
      2021-03-04T10:06:09.725-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(^H^F_sync:unusedSeq:56555067^@^@^@)</ud> docid = <ud>(_sync:unusedSeq:56555067)</ud> Index sg_syncDocs_x1, Bucket default, IndexInstId 13320080993727085165, PartitionId 0
      		 
      2021-03-04T10:06:09.744-08:00 [Fatal] plasmaSlice::handleCommandsWorker: panic detected while processing mutation for operation 1 key = <ud>(^H^F_sync:unusedSeq:56555090^@^@^@)</ud> docid = <ud>(_sync:unusedSeq:56555090)</ud> Index sg_syncDocs_x1, Bucket default, IndexInstId 13320080993727085165, PartitionId 0
      		 
      2021-03-04T10:06:09.767-08:00 [Fatal] goroutine 176366914 [running]:
      		 
      github.com/couchbase/indexing/secondary/logging.(*destination).StackTraceAll(...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:175
      		 
      github.com/couchbase/indexing/secondary/logging.StackTraceAll(0x20ff5f0, 0x2120001)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:317 +0x6d
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker.func1(0xc03c7aded0, 0xc012e01700)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:596 +0x424
      		 
      panic(0x10a1e00, 0xc03c787c70)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      		 
      github.com/couchbase/plasma.(*Shard).raisePanic(...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1011
      		 
      github.com/couchbase/plasma.(*LSSCtx).raiseCorrupted(0xc0062c3040, 0x1488ee0, 0xc03c787c70)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lssctx.go:179 +0x206
      		 
      github.com/couchbase/plasma.(*Plasma).fatalPanic(0xc00a9b0580, 0xc03c7822a0, 0x21)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:1354 +0x293
      		 
      github.com/couchbase/plasma.(*page).lookup(0xc01e86af40, 0xc03c7ad730, 0x7f0c5629c520, 0x0, 0xcbd800)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page.go:766 +0x1f4
      		 
      github.com/couchbase/plasma.(*page).Lookup(0xc01e86af40, 0x7f0c5629c520, 0x0, 0x7f0c53ae8800)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page.go:667 +0x105
      		 
      github.com/couchbase/plasma.(*Writer).Lookup(0xc0193a47b0, 0x7f0c5629c520, 0x18, 0x20, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/writer.go:135 +0x103
      		 
      github.com/couchbase/plasma.(*Writer).LookupKV(0xc0193a47b0, 0xc00be266c0, 0x18, 0x20, 0xc0073ef940, 0xc0073ef978, 0x434e81, 0x12f0150, 0xc0073ef988)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mvcc.go:395 +0x10c
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).deleteSecIndex(0xc012e01700, 0xc00be266c0, 0x18, 0x20, 0xc00be267c0, 0x1d, 0x20, 0x2, 0x0, 0x100000000000000)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:1141 +0x13b
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insertSecIndex(0xc012e01700, 0xc00be267c0, 0x1d, 0x20, 0xc00be266c0, 0x18, 0x20, 0x2, 0x4c2500, 0xc01c295480, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:775 +0xb18
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).insert(0xc012e01700, 0xc00be267c0, 0x1d, 0x20, 0xc00be266c0, 0x18, 0x20, 0x2, 0xc01ea76200, 0xc01c295480, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:733 +0x164
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).handleCommandsWorker(0xc012e01700, 0x2)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:610 +0x58a
      		 
      created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initWriters
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:2980 +0x521

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-40192 compact when marshalling full page over purge ratio

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-47354 BP 7.0.2 MB-40192 - compact when marshalling full page over purge ratio

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-44749
          # Subject Branch Project Status CR V
          148985,1 MB-44749: Add test for gorountine leak unstable plasma Status: NEW 0 -1
          149526,2 MB-44749: Add unit test for concurrent writer and snapshot unstable plasma Status: NEW 0 -1
          147769,2 MB-44749: Comment out evict on log cleaning unstable plasma Status: MERGED +2 +1
          148183,3 MB-44749: Preserve compressed bit in SetOp unstable plasma Status: MERGED +2 +1
          148210,3 MB-44749: Use gCtx in freePtrs for capturing reclaim free stats unstable plasma Status: MERGED +2 +1
          148283,5 MB-44749: Use SetOp for basePage and swapoutDelta unstable plasma Status: MERGED +2 +1
          148309,2 MB-44749: Comment out compact on split and compact on full marshal unstable plasma Status: MERGED +2 +1
          148363,3 MB-44749: Output compress bit in op stringer unstable plasma Status: MERGED +2 +1
          148371,1 MB-44749: Add unit test for high compact conflicts unstable plasma Status: ABANDONED 0 -1
          148509,4 MB-44749: Panic if unable to delete node from skiplist unstable plasma Status: MERGED +2 +1
          148588,1 MB-44749: Add ref counting in sCtx and wCtx unstable plasma Status: MERGED +2 +1
          148603,7 MB-44749: Reset allocCtx in wCtx upon freed unstable plasma Status: MERGED +2 +1
          148770,2 MB-44749: Comment out wCtx and sCtx trimming unstable plasma Status: ABANDONED 0 0
          148785,4 MB-44749: Use concurrent stack instead of concurrent queue unstable plasma Status: MERGED +2 +1
          148787,2 MB-44749: Comment evict on dirty for perioidc eviction unstable plasma Status: MERGED +2 +1
          148895,2 MB-44749: comment out compress and decompress explicitly unstable plasma Status: MERGED +2 +1
          149120,6 MB-44749: Add assertion unstable plasma Status: MERGED +2 +1
          149448,2 MB-44749: Comment out free memory in newSnapshot unstable plasma Status: MERGED +2 +1
          149643,1 MB-44749: Add assertion unstable plasma Status: MERGED +2 +1
          149769,1 MB-44749: Disable newSnapshot assertion unstable plasma Status: MERGED +2 +1
          149781,6 MB-44749: Assert active writer when creating snapshot unstable plasma Status: MERGED +2 +1
          150348,1 Revert "MB-44749: Assert active writer when creating snapshot" unstable plasma Status: ABANDONED +1 -1

          Activity

            People

              jliang John Liang
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are 2 open Gerrit changes

                  PagerDuty