Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45747

cbdocloader race condition

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.1.0
    • 6.6.0, 6.6.1, 6.6.2, Cheshire-Cat
    • tools
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      When running local XDCR race tests with golang race detector, found the following race condition:

      Note that my source tree for backup is currently sitting on e567555cd5a1b91bf81b212f7b6c1d5a65ba2e60
      and gocbcore v9 sitting at 381d7c496fc62a757d5a044396c58fe38c2e99dd

      ==================
      WARNING: DATA RACE
      Write at 0x00c000125118 by goroutine 45:
        github.com/couchbase/backup/couchbase.(*operation).complete()
            /Users/neil.huang/source/couchbase/backup/couchbase/operation.go:118 +0xa9
        github.com/couchbase/backup/couchbase.(*Set).callback()
            /Users/neil.huang/source/couchbase/backup/couchbase/operation.go:195 +0x64
        github.com/couchbase/backup/couchbase.(*Set).callback-fm()
            /Users/neil.huang/source/couchbase/backup/couchbase/operation.go:194 +0x2b
        github.com/couchbase/gocbcore/v9.(*crudComponent).store.func1()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/crudcomponent.go:656 +0x279
        github.com/couchbase/gocbcore/v9.(*memdQRequest).tryCallback()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdqpackets.go:192 +0x1ef
        github.com/couchbase/gocbcore/v9.(*memdClient).resolveRequest()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdclient.go:313 +0x848
        github.com/couchbase/gocbcore/v9.(*memdClient).run.func2()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdclient.go:425 +0x824
       
      Previous write at 0x00c000125118 by goroutine 55:
        github.com/couchbase/backup/couchbase.(*Set).Dispatch()
            /Users/neil.huang/source/couchbase/backup/couchbase/operation.go:168 +0x224
        github.com/couchbase/backup/couchbase.(*MemcachedWorker).dispatchOperation()
            /Users/neil.huang/source/couchbase/backup/couchbase/pool_worker.go:260 +0xa5
        github.com/couchbase/backup/couchbase.(*MemcachedWorker).startSetDel.func1()
            /Users/neil.huang/source/couchbase/backup/couchbase/pool_worker.go:166 +0x31c
       
      Goroutine 45 (running) created at:
        github.com/couchbase/gocbcore/v9.(*memdClient).run()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdclient.go:355 +0x13b
        github.com/couchbase/gocbcore/v9.newMemdClient()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdclient.go:99 +0x3f6
        github.com/couchbase/gocbcore/v9.(*memdClientDialerComponent).dialMemdClient()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdclientdialer_component.go:155 +0x5cb
        github.com/couchbase/gocbcore/v9.(*memdClientDialerComponent).SlowDialMemdClient()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdclientdialer_component.go:90 +0x2a4
        github.com/couchbase/gocbcore/v9.(*kvMux).newKVMuxState.func1()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/kvmux.go:554 +0x10f
        github.com/couchbase/gocbcore/v9.(*memdPipelineClient).Run.func1()
            /Users/neil.huang/source/couchbase/godeps/src/github.com/couchbase/gocbcore/v9/memdpipelineclient.go:220 +0x7d
       
      Goroutine 55 (running) created at:
        github.com/couchbase/backup/couchbase.(*MemcachedWorker).startSetDel()
            /Users/neil.huang/source/couchbase/backup/couchbase/pool_worker.go:155 +0x69
        github.com/couchbase/backup/couchbase.(*MemcachedWorker).Start()
            /Users/neil.huang/source/couchbase/backup/couchbase/pool_worker.go:144 +0x22a
        github.com/couchbase/backup/couchbase.(*MemcachedWorkerPool).Start()
            /Users/neil.huang/source/couchbase/backup/couchbase/pool.go:95 +0x11d
        github.com/couchbase/backup/couchbase.createCallbacks()
            /Users/neil.huang/source/couchbase/backup/couchbase/callbacks.go:53 +0x113
        github.com/couchbase/backup/couchbase.(*Sink).GetDataCallbacks()
            /Users/neil.huang/source/couchbase/backup/couchbase/sink.go:329 +0x3b0
        github.com/couchbase/backup/plan/services/data.(*keyValue).execute()
            /Users/neil.huang/source/couchbase/backup/plan/services/data/key_value.go:99 +0xf0
        github.com/couchbase/backup/plan/services/data.(*keyValue).Execute()
            /Users/neil.huang/source/couchbase/backup/plan/services/data/key_value.go:78 +0x294
        github.com/couchbase/backup/plan/services/data.(*Data).Execute()
            /Users/neil.huang/source/couchbase/backup/plan/services/data/data.go:58 +0x339
        github.com/couchbase/backup/plan/cluster/bucket.(*Bucket).Execute()
            /Users/neil.huang/source/couchbase/backup/plan/cluster/bucket/bucket.go:185 +0x12b6
        github.com/couchbase/backup/plan/cluster.(*Cluster).Execute.func1()
            /Users/neil.huang/source/couchbase/backup/plan/cluster/cluster.go:84 +0x1b9
        github.com/couchbase/backup/plan/cluster.(*Cluster).Execute()
            /Users/neil.huang/source/couchbase/backup/plan/cluster/cluster.go:93 +0xfa
        github.com/couchbase/backup/plan.(*Plan).Execute()
            /Users/neil.huang/source/couchbase/backup/plan/plan.go:76 +0x170
        github.com/couchbase/backup/transferable.(*transferable).Execute()
            /Users/neil.huang/source/couchbase/backup/transferable/transferable.go:39 +0xdb
        github.com/couchbase/backup/cmd/jsonctx.(*JSONContext).Run()
            /Users/neil.huang/source/couchbase/backup/cmd/jsonctx/common.go:123 +0xbc6
        main.(*docloaderCTX).run()
            backup/cmd/cbdocloader/main.go:33 +0x164
        main.(*docloaderCTX).run-fm()
            backup/cmd/cbdocloader/main.go:27 +0x4a
        github.com/couchbase/cbflag.(*Command).parseFlags()
            /Users/neil.huang/source/couchbase/build/gopkg/go-1.16.2/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20201020130537-3393038a55d7/command.go:241 +0x23d5
        github.com/couchbase/cbflag.(*Command).parse()
            /Users/neil.huang/source/couchbase/build/gopkg/go-1.16.2/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20201020130537-3393038a55d7/command.go:92 +0x25e
        github.com/couchbase/cbflag.(*CLI).Parse()
            /Users/neil.huang/source/couchbase/build/gopkg/go-1.16.2/pkg/mod/github.com/couchbase/cbflag@v0.0.0-20201020130537-3393038a55d7/cli.go:59 +0x3e5
        main.main()
            backup/cmd/cbdocloader/main.go:142 +0x1dad
      ==================
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              thuan Thuan Nguyen
              neil.huang Neil Huang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty