Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29800

Rebalance Failures & Plasma panics seen in 5.1.1 system test run - "panic: duplicate index node"

    XMLWordPrintable

Details

    Description

      Build : 5.1.1-5720

      The indexer logs on one of the index nodes - 172.23.96.148 shows a lot of panic messages like the following couple of them :

      2018-05-22T18:09:35.033-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
      		 
               Index: 17689712609126445135 Partition: PartitionId: 0 Endpoints: [:9105]
      		 
      2018-05-22T18:09:36.266-07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 17689712609126445135 Recovering from recovery point ..
      		 
      2018-05-22T18:09:36.303-07:00 [Info] plasmaSlice::doRecovery SliceId 0 IndexInstId 17689712609126445135 Warmup took 1.106382471s
      		 
      2018-05-22T18:09:36.303-07:00 [Info] plasmaSlice:NewplasmaSlice Created New Slice Id 0 IndexInstId 17689712609126445135 WriterThreads 8
      		 
      2018-05-22T18:09:36.303-07:00 [Info] Indexer::initPartnInstance Initialized Slice:
      		 
               Index: 17689712609126445135 Slice: SliceId: 0 File: /data/@2i/CUSTOMER_o1_result_17689712609126445135_0.index Index: 17689712609126445135
      		 
      2018-05-22T18:09:36.303-07:00 [Info] Indexer::initPartnInstance Initialized Partition:
      		 
               Index: 2308736630918829209 Partition: PartitionId: 0 Endpoints: [:9105]
      		 
      panic: found missing page
      		 
       
      		 
      goroutine 800 [running]:
      		 
      panic(0xd01400, 0xc4255894a0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc423639df8 sp=0xc423639d68
      		 
      github.com/couchbase/plasma.(*Plasma).doRecovery.func2(0xdd11c0, 0x7fc000f9ed40, 0x0, 0x0, 0xffffffffffffffff, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:646 +0x220 fp=0xc423639e98 sp=0xc423639df8
      		 
      github.com/couchbase/plasma.(*Plasma).VisitPartition(0xc4237f8580, 0x0, 0x0, 0xffffffffffffffff, 0xc42a635920, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:71 +0x2e7 fp=0xc423639f20 sp=0xc423639e98
      		 
      github.com/couchbase/plasma.(*Plasma).PageVisitor.func1(0xc425589480, 0xc425589490, 0x1, 0x1, 0xc4237f8580, 0xc42a635920, 0x0, 0x0, 0xffffffffffffffff)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:40 +0x89 fp=0xc423639f68 sp=0xc423639f20
      		 
      runtime.goexit()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc423639f70 sp=0xc423639f68
      		 
      created by github.com/couchbase/plasma.(*Plasma).PageVisitor
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/page_visitor.go:41 +0x1ae


      panic: invalid last page
      		 
       
      		 
      goroutine 823 [running]:
      		 
      panic(0xd01400, 0xc42ed521d0)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc42f68b870 sp=0xc42f68b7e0
      		 
      github.com/couchbase/plasma.(*Plasma).doRecovery(0xc4237f8580, 0xc4306c9360, 0x21)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:662 +0x41d fp=0xc42f68b938 sp=0xc42f68b870
      		 
      github.com/couchbase/plasma.New(0xc8, 0x190, 0x19, 0x4, 0xf30198, 0xf30160, 0xf301a0, 0xc42b4e7d10, 0xc42b4e7d20, 0xf30160, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:429 +0xf9a fp=0xc42f68bde8 sp=0xc42f68b938
      		 
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0xc42a004850, 0xc429fec600, 0xc42a004840, 0xc426d6bb00, 0xc42a004860)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:233 +0x91 fp=0xc42f68bf78 sp=0xc42f68bde8
      		 
      runtime.goexit()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc42f68bf80 sp=0xc42f68bf78
      		 
      created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:249 +0x15e2
      		 
      [goport(/opt/couchbase/bin/indexer)] 2018/05/22 18:09:40 child process exited with status 134

      The indexer process continously crashes on this node.

      2i data backup : /root/backup_2i_data on 172.23.96.148

      Note : We havent seen this issue during the system test runs done in 5.0.0 or 5.1.0, or even 5.5.0

      Attachments

        1. bug.tgz
          38.59 MB
        2. tools.tgz
          5 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-30242 CLONE (Backport MB-29800) - Rebalance Failures & Plasma panics seen in 5.1.1 system test run - "panic: duplicate index node"

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sarath Sarath Lakshman
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty