Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-18887

goxdcr panic during rebalance in [goroutine created by TopologyChangeDetectorSvc]

    XMLWordPrintable

Details

    Description

      During longevity test, performed failover + rebalance and noticed xdcr stream stopped with errors in logs:

      Service 'goxdcr' exited with status 1. Restarting. Messages: runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc8202547c0 sp=0xc8202547b8
      created by runtime.gcBgMarkStartWorkers
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/mgc.go:1239 +0x93
      [goport] 2016/03/24 18:30:25 /opt/couchbase/bin/goxdcr terminated: signal: aborted (core dumped)

      validateSourceTopology...
      panic: runtime error: index out of range

      goroutine 464435 [running]:
      runtime.gopanic(0x9e04c0, 0xc8200160a0)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/panic.go:461 +0x3e6 fp=0xc8288b1550 sp=0xc8288b14d0
      runtime.panicindex()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/panic.go:12 +0x49 fp=0xc8288b1578 sp=0xc8288b1550
      github.com/couchbase/go-couchbase.(*Bucket).GetVBmap(0xc826eebdc0, 0xc82a9aaf00, 0x3, 0x4, 0x7, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/watson-unix/godeps/src/github.com/couchbase/go-couchbase/pools.go:247 +0x4c3 fp=0xc8288b16e8 sp=0xc8288b1578
      github.com/couchbase/goxdcr/service_impl.(*ClusterInfoSvc).GetServerVBucketsMap(0xc8200fc030, 0x7f12a4170750, 0xc820100620, 0xc82064b580, 0x7, 0x0, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/goxdcr/service_impl/cluster_info_service.go:78 +0x297 fp=0xc8288b17a0 sp=0xc8288b16e8
      github.com/couchbase/goxdcr/pipeline_utils.GetSourceVBMap(0x7f12a41ae4b8, 0xc8200fc030, 0x7f12a41ae508, 0xc820100620, 0xc82064b580, 0x7, 0xc823158760, 0xc825bf4a00, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_utils/pipeline_utils.go:38 +0xdf fp=0xc8288b19a8 sp=0xc8288b17a0
      github.com/couchbase/goxdcr/pipeline_svc.(*TopologyChangeDetectorSvc).validateSourceTopology(0xc820310680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/topology_change_detector.go:293 +0x2d4 fp=0xc8288b1b98 sp=0xc8288b19a8
      github.com/couchbase/goxdcr/pipeline_svc.(*TopologyChangeDetectorSvc).validate(0xc820310680, 0xbd0000)
      /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/topology_change_detector.go:127 +0x55 fp=0xc8288b1d80 sp=0xc8288b1b98
      github.com/couchbase/goxdcr/pipeline_svc.(*TopologyChangeDetectorSvc).watch(0xc820310680, 0xc8286f3f80, 0xc823158770)
      /home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/topology_change_detector.go:121 +0xd21 fp=0xc8288b1f78 sp=0xc8288b1d80
      runtime.goexit()
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.5.2/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc8288b1f80 sp=0xc8288b1f78
      created by github.com/couchbase/goxdcr/pipeline_svc.(*TopologyChangeDetectorSvc).Start
      /home/couchbase/jenkins/workspace/watson-unix/gopr

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              yu Yu Sui (Inactive)
              tommie Tommie McAfee (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty