Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30801

[System Test] Plasma crash : Corrupted item detected

    XMLWordPrintable

Details

    Description

      Build : 5.1.2-6016
      Iteration : 1
      Scale : 3
      Test : -test tests/2i/test_idx_rebalance_replica_spock.yml -scope tests/2i/scope_idx_rebalance_replica_12nodes.yml
      Cluster : Centos2

      Seeing plasma crash on an indexer node while another indexer node failed over and removed from the cluster. The stack trace is similar to what was seen in MB-30492. This bug could also be related to MB-29952

      Here is the panic msg seen on 172.23.96.254 :

      2018-08-07T07:49:17.227-07:00 [Info] default/default_result/Backstore Plasma: logCleaner: starting... frag 31, data: 119312344, used: 174093758 log:(1535029530 - 1710063119)
      		 
      2018-08-07T07:49:17.234-07:00 [Info] plasmaSlice::OpenSnapshot SliceId 0 IndexInstId 364261172130525603 Creating New Snapshot SnapshotInfo: count:2600642 committed:false
      		 
      2018-08-07T07:49:17.234-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 364261172130525603 PartitionId: 0 SliceId: 0 Crc64: 12939052796789740186 (SnapshotInfo: count:2600642 committed:false) SnapCreateDur 8.521µs SnapOpenDur 46.944µs
      		 
      2018-08-07T07:49:17.245-07:00 [Info] plasmaSlice::OpenSnapshot SliceId 0 IndexInstId 10856253011618663302 Creating New Snapshot SnapshotInfo: count:707118 committed:false
      		 
      2018-08-07T07:49:17.245-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 10856253011618663302 PartitionId: 0 SliceId: 0 Crc64: 11492456263764330149 (SnapshotInfo: count:707118 committed:false) SnapCreateDur 8.272µs SnapOpenDur 58.486µs
      		 
      2018-08-07T07:49:17.247-07:00 [Info] default/default_result/Mainstore Plasma: logCleaner: completed... frag 30, data: 151747691, used: 219922561, relocated: 69, retries: 0, skipped: 335 log:(881737763 - 1102335312)
      		 
      2018-08-07T07:49:17.255-07:00 [Info] plasmaSlice::OpenSnapshot SliceId 0 IndexInstId 364261172130525603 Creating New Snapshot SnapshotInfo: count:2600645 committed:false
      		 
      2018-08-07T07:49:17.255-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 364261172130525603 PartitionId: 0 SliceId: 0 Crc64: 12939052796789740186 (SnapshotInfo: count:2600645 committed:false) SnapCreateDur 8.473µs SnapOpenDur 58.681µs
      		 
      panic: Corrupted item detected:
      		 
       00000000  00 19 00 be 03 25 00 00  c0 13 00 00 00 30 33 44  |.....%.......03D|
      		 
       
      		 
       
      		 
      goroutine 4482806 [running]:
      		 
      panic(0xd06640, 0xc4400e5760)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/panic.go:500 +0x1a1 fp=0xc463f05788 sp=0xc463f056f8
      		 
      github.com/couchbase/plasma.(*item).checkInvalidHeader(0xc43af32003)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:214 +0x188 fp=0xc463f05818 sp=0xc463f05788
      		 
      github.com/couchbase/plasma.(*item).Key(0xc43af32003, 0x0, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:98 +0x5c fp=0xc463f05830 sp=0xc463f05818
      		 
      github.com/couchbase/plasma.cmpItem(0x7fee46d06c60, 0xc43af32003, 0x7fee9a7d29c000)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/item.go:179 +0xaa fp=0xc463f05890 sp=0xc463f05830
      		 
      github.com/couchbase/nitro/skiplist.compare(0xf36d20, 0x7fee46d06c60, 0xc43af32003, 0xc44ba21800)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/item.go:32 +0x88 fp=0xc463f058b8 sp=0xc463f05890
      		 
      github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0xc42c52d860, 0xc43af32003, 0xf36d20, 0xc42ad6adb0, 0xc42c52d878, 0xc44ba21880)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:232 +0x190 fp=0xc463f05930 sp=0xc463f058b8
      		 
      github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0xc42c52d860, 0xc43af32003, 0xf36d20, 0xc42ad6adb0, 0xc42c52d878, 0xc44ba21880, 0x7feec5bf5bc0, 0xc4306f3720)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x57 fp=0xc463f05970 sp=0xc463f05930
      		 
      github.com/couchbase/plasma.newStoreContext.func1(0xc43af32003, 0xc42fb12600, 0x7fec21a383c0, 0x16a0120)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/pagetable.go:88 +0xae fp=0xc463f059c0 sp=0xc463f05970
      		 
      github.com/couchbase/plasma.(*Plasma).CleanLSS.func1(0x5b940faa, 0x5b9415c8, 0xc44d356000, 0x61a, 0x21bf8, 0xd93d00, 0x0, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss_cleaner.go:87 +0x297 fp=0xc463f05ae0 sp=0xc463f059c0
      		 
      github.com/couchbase/plasma.(*lsStore).RunCleaner.func1(0x5b940faa, 0xc44d356000, 0x61a, 0x21bf8, 0x0, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss.go:320 +0x6d fp=0xc463f05b50 sp=0xc463f05ae0
      		 
      github.com/couchbase/plasma.(*lsStore).visitor(0xc429a401e0, 0x5b8d0851, 0x65ed7e0f, 0xc44c1b05a0, 0xc43ab6ae60, 0xc44c1b0540, 0x2)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss.go:350 +0x167 fp=0xc463f05bc0 sp=0xc463f05b50
      		 
      github.com/couchbase/plasma.(*lsStore).RunCleaner(0xc429a401e0, 0xc44e086500, 0xc43ab6ae60, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss.go:334 +0x133 fp=0xc463f05c28 sp=0xc463f05bc0
      		 
      github.com/couchbase/plasma.(*Plasma).CleanLSS(0xc423672580, 0xc442457a30, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss_cleaner.go:147 +0x57e fp=0xc463f05e50 sp=0xc463f05c28
      		 
      github.com/couchbase/plasma.(*Plasma).lssCleanerDaemon(0xc423672580)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/lss_cleaner.go:192 +0x179 fp=0xc463f05f48 sp=0xc463f05e50
      		 
      runtime.goexit()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.3/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc463f05f50 sp=0xc463f05f48
      		 
      created by github.com/couchbase/plasma.New
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:439 +0x1115

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jliang John Liang
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty