Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-23971

MOI test failed in GSI CI run after moving to Go 1.8.1

    XMLWordPrintable

Details

    Description

      After below is fixed, please revert:
      08c02dbe Revert "MB-23969: Move to 1.8.1"
      49c84b80 MB-23920: Refresh SSL certificate in 1.7.3 way

      After moving indexer to Go 1.8.1, CI run failed in MOI storage mode in test case TestBuildDeferredAnotherBuilding. From indexer logs around the time of test failure, two crashes were found:

      #1:

      2017-04-18T11:49:28.570+05:30 [Info] TK StreamBegin INIT_STREAM default 888 263529442115825 0
      runtime: pointer 0xc42887b8c0 to unused region of span idx=0x443d span.base()=0xc42ad52000 span.limit=0xc42ad54000 span.state=1
      fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

      runtime stack:
      runtime.throw(0xe84977, 0x3e)
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/panic.go:596 +0x95 fp=0x7fe796ffdd18 sp=0x7fe796ffdcf8
      runtime.heapBitsForObject(0xc42887b8c0, 0x0, 0x0, 0x0, 0x0, 0x7fe700000000, 0x0, 0x0)
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mbitmap.go:433 +0x2bb fp=0x7fe796ffdd70 sp=0x7fe796ffdd18
      runtime.shade(0xc42887b8c0)
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mgcmark.go:1340 +0x41 fp=0x7fe796ffddc8 sp=0x7fe796ffdd70
      runtime.gcmarkwb_m(0x7fe794022658, 0x7fe78422c540)
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mbarrier.go:154 +0xea fp=0x7fe796ffdde0 sp=0x7fe796ffddc8
      runtime.writebarrierptr_prewrite1.func1()
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mbarrier.go:188 +0x64 fp=0x7fe796ffde00 sp=0x7fe796ffdde0
      runtime.systemstack(0xc42001f300)
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:327 +0x79 fp=0x7fe796ffde08 sp=0x7fe796ffde00
      runtime.mstart()
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:1132 fp=0x7fe796ffde10 sp=0x7fe796ffde08

      goroutine 257213 [running]:
      runtime.systemstack_switch()
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:281 fp=0xc4251caa18 sp=0xc4251caa10
      runtime.writebarrierptr_prewrite1(0x7fe794022658, 0x7fe78422c540)
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mbarrier.go:189 +0xbf fp=0xc4251caa58 sp=0xc4251caa18
      runtime.writebarrierptr(0x7fe794022658, 0x7fe78422c540)
      /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mbarrier.go:211 +0x4d fp=0xc4251caa90 sp=0xc4251caa58
      github.com/couchbase/indexing/secondary/memdb/skiplist.(*Skiplist).Insert3(0xc424b34000, 0x7fe77c14dc40, 0xc4201b1300, 0xc4201b1320, 0xc43943b050, 0x4, 0x7fe77c14dc00, 0xc42219d6e8, 0x0, 0x0)
      /opt/build/goproj/src/github.com/couchbase/indexing/secondary/memdb/skiplist/skiplist.go:235 +0x563 fp=0xc4251cab00 sp=0xc4251caa90
      github.com/couchbase/indexing/secondary/memdb/skiplist.(*Skiplist).Insert2(0xc424b34000, 0x7fe77c14dc40, 0xc4201b1300, 0xc4201b1320, 0xc43943b050, 0xc4251cabb8, 0xc42219d6e8, 0x41608d, 0xc42ef7f560)
      /opt/build/goproj/src/github.com/couchbase/indexing/secondary/memdb/skiplist/skiplist.go:210 +0x91 fp=0xc4251cab60 sp=0xc4251cab00
      github.com/couchbase/indexing/secondary/memdb.(*Writer).Put2(0xc42219d680, 0xc431caa000, 0x25, 0x3902, 0x18)
      /opt/build/goproj/src/github.com/couchbase/indexing/secondary/memdb/memdb.go:198 +0x10b fp=0xc4251cabd8 sp=0xc4251cab60

       

       

      #2

      2017-04-18T12:01:10.179+05:30 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 16682140985260369687 PartitionId: 0 SliceId: 0 Crc64: 8357976057330050722 (SnapshotInfo: count:58367 committed:false) SnapCreateDur 19.263µs SnapOpenDur 51.384µs

      runtime: nelems=14 nfree=11 nalloc=3 previous allocCount=2 nfreed=65535

      fatal error: sweep increased allocation count

       

      runtime stack:

      runtime.throw(0xe6eda9, 0x20)

              /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/panic.go:596 +0x95 fp=0x7eff193feca8 sp=0x7eff193fec88

      runtime.(*mspan).sweep(0x7eff284ef0b0, 0x7eff284ef000, 0xc420001040)

              /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mgcsweep.go:288 +0x786 fp=0x7eff193fed78 sp=0x7eff193feca8

      runtime.sweepone(0x4379e6)

              /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mgcsweep.go:110 +0x1ad fp=0x7eff193fede0 sp=0x7eff193fed78

      runtime.gosweepone.func1()

              /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/mgcsweep.go:125 +0x2b fp=0x7eff193fee00 sp=0x7eff193fede0

      runtime.systemstack(0xc42001f300)

              /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:327 +0x79 fp=0x7eff193fee08 sp=0x7eff193fee00

      runtime.mstart()

              /home/buildbot/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:1132 fp=0x7eff193fee10 sp=0x7eff193fee08

       

      The above two errors occured totally twice. 

      Run link: http://ci2i-unstable.northscale.in/gsi-memdb-18.04.2017-10.46.fail.html

      Server log link: http://ci2i-unstable.northscale.in/logs-18.04.2017-10.46.tar.gz

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jliang John Liang
              prathibha Prathibha Bisarahalli (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty