Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29800

Rebalance Failures & Plasma panics seen in 5.1.1 system test run - "panic: duplicate index node"

    XMLWordPrintable

Details

    Description

      Build : 5.1.1-5720

      The indexer logs on one of the index nodes - 172.23.96.148 shows a lot of panic messages like the following couple of them :

      2018-05-22T18:09:35.033-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
               Index: 17689712609126445135 Partition: PartitionId: 0 Endpoints: [:9105]
      2018-05-22T18:09:36.266-07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 17689712609126445135 Recovering from recovery point ..
      2018-05-22T18:09:36.303-07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 17689712609126445135 Warmup took 1.106382471s
      2018-05-22T18:09:36.303-07:00 [Info] plasmaSlice:NewplasmaSlice Created New Slice Id 0 IndexInstId 17689712609126445135 WriterThreads 8
      2018-05-22T18:09:36.303-07:00 [Info] Indexer::initPartnInstance Initialized Slice:
               Index: 17689712609126445135 Slice: SliceId: 0 File: /data/@2i/CUSTOMER_o1_result_17689712609126445135_0.index Index: 17689712609126445135
      2018-05-22T18:09:36.303-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
               Index: 2308736630918829209 Partition: PartitionId: 0 Endpoints: [:9105]
      panic: found missing page
       
      goroutine 800 [running]:
      panic(0xd01400, 0xc4255894a0)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc423639df8 sp=0xc423639d68
      github.com/couchbase/plasma.(*Plasma).doRecovery.func2(0xdd11c0, 0x7fc000f9ed40, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:646 +0x220 fp=0xc423639e98 sp=0xc423639df8
      github.com/couchbase/plasma.(*Plasma).VisitPartition(0xc4237f8580, 0x0, 0x0, 0xffffffffffffffff, 0xc42a635920, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:71 +0x2e7 fp=0xc423639f20 sp=0xc423639e98
      github.com/couchbase/plasma.(*Plasma).PageVisitor.func1(0xc425589480, 0xc425589490, 0x1, 0x1, 0xc4237f8580, 0xc42a635920, 0x0, 0x0, 0xffffffffffffffff)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:40 +0x89 fp=0xc423639f68 sp=0xc423639f20
      runtime.goexit()
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc423639f70 sp=0xc423639f68
      created by github.com/couchbase/plasma.(*Plasma).PageVisitor
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:41 +0x1ae
      

      panic: invalid last page
       
      goroutine 823 [running]:
      panic(0xd01400, 0xc42ed521d0)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc42f68b870 sp=0xc42f68b7e0
      github.com/couchbase/plasma.(*Plasma).doRecovery(0xc4237f8580, 0xc4306c9360, 0x21)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:662 +0x41d fp=0xc42f68b938 sp=0xc42f68b870
      github.com/couchbase/plasma.New(0xc8, 0x190, 0x19, 0x4, 0xf30198, 0xf30160, 0xf301a0, 0xc42b4e7d10, 0xc42b4e7d20, 0xf30160, ...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:429 +0xf9a fp=0xc42f68bde8 sp=0xc42f68b938
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0xc42a004850, 0xc429fec600, 0xc42a004840, 0xc426d6bb00, 0xc42a004860)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:233 +0x91 fp=0xc42f68bf78 sp=0xc42f68bde8
      runtime.goexit()
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42f68bf80 sp=0xc42f68bf78
      created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:249 +0x15e2
      [goport(/opt/couchbase/bin/indexer)] 2018/05/22 18:09:40 child process exited with status 134
      

      The indexer process continously crashes on this node.

      2i data backup : /root/backup_2i_data on 172.23.96.148

      Note : We havent seen this issue during the system test runs done in 5.0.0 or 5.1.0, or even 5.5.0

      Attachments

        1. bug.tgz
          38.59 MB
        2. tools.tgz
          5 kB

        Issue Links

          For Gerrit Dashboard: MB-29800
          # Subject Branch Project Status CR V

          Activity

            Build couchbase-server-6.5.0-1031 contains plasma commit f9be03b with commit message:
            MB-29800 rollback: Do not allow lss cleaner and mvcc purger during rollback

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1031 contains plasma commit f9be03b with commit message: MB-29800 rollback: Do not allow lss cleaner and mvcc purger during rollback

            Build couchbase-server-6.5.0-1031 contains plasma commit a6ef214 with commit message:
            MB-29800 config: Enable mvcc purging by default

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1031 contains plasma commit a6ef214 with commit message: MB-29800 config: Enable mvcc purging by default

            Build couchbase-server-6.5.0-1031 contains plasma commit 42eeb86 with commit message:
            MB-29800 mvcc_purger: Do not compact pages with in-progress merge

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1031 contains plasma commit 42eeb86 with commit message: MB-29800 mvcc_purger: Do not compact pages with in-progress merge

            Build couchbase-server-6.5.0-1031 contains plasma commit 4c48b06 with commit message:
            MB-29800 page: Add page mutability check for page operations

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1031 contains plasma commit 4c48b06 with commit message: MB-29800 page: Add page mutability check for page operations

            Build couchbase-server-6.5.0-1091 contains plasma commit c74f295 with commit message:
            MB-29800 Merge commit '97a3fdc' into HEAD

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1091 contains plasma commit c74f295 with commit message: MB-29800 Merge commit '97a3fdc' into HEAD

            People

              sarath Sarath Lakshman
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty