Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24886

panic: snappy: corrupt input

    XMLWordPrintable

Details

    • Task
    • Resolution: Duplicate
    • Major
    • 5.5.0
    • 5.0.0
    • secondary-index
    • Enterprise Edition 5.0.0 build 3083

    Description

      Steps to Repro(It must be noted tests were run on a vagrant VM).

      1) Created a 3 node cluster - kv-index1-n1ql
      2) Pumped in lot of data and created multiple secondary indexes.
      3) added one more indexer to the cluster and did a gsi rebalance(lets say indexer2). All the indexes will still be in indexer1 as per gsi rebalance design.
      4) started creating docs on the bucket until indexer1 went into paused state.
      5) Now tried to rebalance out indexer2.
      6) GSI rebalance failed.(which is expected)

      However all the indexes on indexer1 got stuck in ready-warmup-ready loop after gsi rebalance. This happened maybe around 10 times. I was also seeing projector crashes along with indexer crashes. At this point my laptop hung and I had to restart it. After restart I manually restarted the vm's from virtual box. I started seeing the following panic in indexer

      panic: snappy: corrupt input

      goroutine 1238 [running]:
      panic(0xcf0060, 0xc42012f020)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc445dc3598 sp=0xc445dc3508
      github.com/couchbase/nitro/plasma.(*wCtx).decompress(0xc42aa88680, 0xc447a94002, 0x2810, 0x338da, 0xc43fcc67c0, 0x2812, 0x338dc, 0x119d0f1)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:695 +0x20c fp=0xc445dc3620 sp=0xc445dc3598
      github.com/couchbase/nitro/plasma.(*page).unmarshalDelta(0xc444f7f000, 0xc447a94002, 0x2810, 0x338da, 0xc42aa88680, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/page.go:962 +0xaf fp=0xc445dc3728 sp=0xc445dc3620
      github.com/couchbase/nitro/plasma.(*page).Unmarshal(0xc444f7f000, 0xc447a94002, 0x2810, 0x338da, 0xc42aa88680)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/page.go:958 +0x53 fp=0xc445dc3770 sp=0xc445dc3728
      github.com/couchbase/nitro/plasma.(*Plasma).doRecovery.func1(0x119d0ed, 0xc447a94000, 0x2812, 0x338dc, 0x0, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:511 +0xda fp=0xc445dc3878 sp=0xc445dc3770
      github.com/couchbase/nitro/plasma.(*lsStore).visitor(0xc43fbb7200, 0x1017d0, 0x129b394, 0xc43fcc67a0, 0xc43fcc6780, 0xd4bc40, 0xc41ed8a601)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/lss.go:309 +0x167 fp=0xc445dc38e8 sp=0xc445dc3878
      github.com/couchbase/nitro/plasma.(*lsStore).Visitor(0xc43fbb7200, 0xc43fcc67a0, 0xc43fcc6780, 0xc4476eb000, 0x1000)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/lss.go:298 +0x9d fp=0xc445dc3938 sp=0xc445dc38e8
      github.com/couchbase/nitro/plasma.(*Plasma).doRecovery(0xc43cd07b80, 0xc444eb30e0, 0x21)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:557 +0x231 fp=0xc445dc3a00 sp=0xc445dc3938
      github.com/couchbase/nitro/plasma.New(0x1e, 0x12c, 0x5, 0x4, 0xede030, 0xede000, 0xede038, 0xc445122a00, 0xc445122a10, 0xede000, ...)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/plasma/plasma.go:366 +0xe25 fp=0xc445dc3e08 sp=0xc445dc3a00
      github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func2(0xc4451229d0, 0xc440f6ec00, 0xc4451229c0, 0xc43e246700, 0xc4451229f0)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:238 +0x91 fp=0xc445dc3f68 sp=0xc445dc3e08
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc445dc3f70 sp=0xc445dc3f68
      created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:250 +0x14dc

      Logs attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sarath Sarath Lakshman
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty