Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25175

Build Sanity : Regression in test XDCRFilterTests.test_xdcr_with_filter. Panic seen in XDCR log

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Build : 5.0.0-3261 and 5.0.0-3262

      A failure in the XDCRFilterTests.test_xdcr_with_filter sanity test is seen across multiple platforms, however not all. The test reports panic in the xdcr log.

      Here's the excerpt from the test log (the test checks for panics in the xdcr logs)

      [2017-07-08 13:18:14,267] - [xdcrnewbasetests:3302] INFO - panic: runtime error: invalid memory address or nil pointer dereference
      		 
      [2017-07-08 13:18:14,267] - [xdcrnewbasetests:3302] INFO - [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x837757]
      		 
      [2017-07-08 13:18:14,267] - [xdcrnewbasetests:3302] INFO - 
      [2017-07-08 13:18:14,267] - [xdcrnewbasetests:3302] INFO - goroutine 4068 [running]:
      		 
      [2017-07-08 13:18:14,267] - [xdcrnewbasetests:3302] INFO - panic(0x91d4e0, 0xc34960)
      		 
      [2017-07-08 13:18:14,267] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/panic.go:531 +0x1cf fp=0xc4202178a0 sp=0xc420217808
      		 
      [2017-07-08 13:18:14,267] - [xdcrnewbasetests:3302] INFO - runtime.panicmem()
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/panic.go:63 +0x5e fp=0xc4202178c0 sp=0xc4202178a0
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - runtime.sigpanic()
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/signal_unix.go:290 +0x29f fp=0xc420217910 sp=0xc4202178c0
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - github.com/couchbase/goxdcr/pipeline_svc.(*CheckpointManager).doCheckpoint(0xc420162900, 0x12ca0390, 0xc4201c9380, 0xc4201c98c0, 0xc4208371d0, 0x0, 0x0)
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/checkpoint_manager.go:1246 +0x9e7 fp=0xc420217d28 sp=0xc420217910
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - github.com/couchbase/goxdcr/pipeline_svc.(*CheckpointManager).performCkpt_internal(0xc420162900, 0xc420146600, 0x66, 0x80, 0xc4201cb1f0, 0xc4204b12e0, 0x0, 0xc4201c9380, 0xc4201c98c0, 0xc4208371d0)
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/checkpoint_manager.go:1095 +0x438 fp=0xc420217f90 sp=0xc420217d28
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - runtime.goexit()
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420217f98 sp=0xc420217f90
      		 
      [2017-07-08 13:18:14,268] - [xdcrnewbasetests:3302] INFO - created by github.com/couchbase/goxdcr/pipeline_svc.(*CheckpointManager).PerformCkpt
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/pipeline_svc/checkpoint_manager.go:1042 +0x4eb
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - goroutine 1 [chan receive]:
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - runtime.gopark(0x9dc418, 0xc420126b38, 0x9b485f, 0xc, 0xc42019fc17, 0x3)
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:271 +0x13a fp=0xc42019dc80 sp=0xc42019dc50
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - runtime.goparkunlock(0xc420126b38, 0x9b485f, 0xc, 0xc42005a817, 0x3)
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:277 +0x5e fp=0xc42019dcc0 sp=0xc42019dc80
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - runtime.chanrecv(0x8ecca0, 0xc420126ae0, 0x0, 0x1, 0x270e)
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/chan.go:513 +0x371 fp=0xc42019dd60 sp=0xc42019dcc0
      		 
      [2017-07-08 13:18:14,269] - [xdcrnewbasetests:3302] INFO - runtime.chanrecv1(0x8ecca0, 0xc420126ae0, 0x0)
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/chan.go:395 +0x35 fp=0xc42019dd98 sp=0xc42019dd60
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - main.main()
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/goxdcr/main/main.go:172 +0xbdc fp=0xc42019df88 sp=0xc42019dd98
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - runtime.main()
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/proc.go:185 +0x20a fp=0xc42019dfe0 sp=0xc42019df88
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - runtime.goexit()
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42019dfe8 sp=0xc42019dfe0
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - goroutine 17 [syscall, locked to thread]:
      		 
      [2017-07-08 13:18:14,270] - [xdcrnewbasetests:3302] INFO - runtime.goexit()
      		 
      [2017-07-08 13:18:14,271] - [xdcrnewbasetests:3302] INFO - 	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.1/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42003efe8 sp=0xc42003efe0
      		 
      [2017-07-08 13:18:14,271] - [xdcrnewbasetests:3302] INFO - 
      [2017-07-08 13:18:14,271] - [xdcrnewbasetests:3302] INFO - goroutine 2 [force gc (idle)]:
      		 
      [2017-07-08 13:18:14,271] - [xdcrnewbasetests:3302] INFO - runtime.gopark(0x9dc418, 0xc43660, 0x9b6774, 0xf, 0x9dc314, 0x1)

      Test : ./testrunner -i node_conf.ini get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False -t xdcr.filterXDCR.XDCRFilterTests.test_xdcr_with_filter,items=1000,pause=C1:C2,rdirection=bidirection,ctopology=chain,default@C1=filter_expression:C1-key-1,default@C2=filter_expression:C2-key-1,update=C1,delete=C1,demand_encryption=1

      This test worked fine till 5.0.0-3259, and then started regressing. The following commit that went into 5.0.0-3260 looks like what broke it :

      • Commit: 52b1e7f6e66b22240802bf834358529086234d4d (in build: 3260)
        Author: Neil Huang
        MB-24562 - checkpoint manager records race conditions

      Change-Id: I19475556d9168f7ed6f53bf4e3149dc66a04a8db
      Reviewed-on: http://review.couchbase.org/78931
      Reviewed-by: John Liang
      Reviewed-by: Yu Sui
      Reviewed-by: Neil Huang
      Tested-by: Neil Huang

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            yu Yu Sui (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty