Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-2448

[3.0.4 Backport] Background manager cluster aware terminator race condition

    XMLWordPrintable

Details

    • CBG Sprint 106
    • 1

    Description

      A race condition is in the background manager cluster aware code to do with the terminator.

      A way to replicate this race locally is to run TestAttachmentCompactionAPI with race, and possibly multiple times. The race seems to happen when the second compaction is started, after the previous one completed however this will need more investigation.

      The race stack:

      WARNING: DATA RACE
      Write at 0x00c0002d47b8 by goroutine 37:
        github.com/couchbase/sync_gateway/db.(*BackgroundManager).markStart()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/background_mgr.go:202 +0x604
        github.com/couchbase/sync_gateway/db.(*BackgroundManager).Start()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/background_mgr.go:96 +0x66
        github.com/couchbase/sync_gateway/rest.(*handler).handleCompact()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/api.go:159 +0xb2b
        github.com/couchbase/sync_gateway/rest.(*handler).invoke()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/handler.go:454 +0x38b9
        github.com/couchbase/sync_gateway/rest.makeHandler.func1()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/handler.go:126 +0xfb
        net/http.HandlerFunc.ServeHTTP()
            /Users/isaaclambat/sdk/go1.17.11/src/net/http/server.go:2047 +0x4d
        github.com/gorilla/mux.(*Router).ServeHTTP()
            /Users/isaaclambat/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x366
        github.com/couchbase/sync_gateway/rest.wrapRouter.func1()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/routing.go:346 +0x8f0
        net/http.HandlerFunc.ServeHTTP()
            /Users/isaaclambat/sdk/go1.17.11/src/net/http/server.go:2047 +0x4d
        github.com/couchbase/sync_gateway/rest.(*RestTester).SendAdminRequest()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/utilities_testing.go:605 +0x451
        github.com/couchbase/sync_gateway/rest.TestAttachmentCompactionAPI()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/attachment_compaction_api_test.go:82 +0xa0c
        testing.tRunner()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1259 +0x22f
        testing.(*T).Run·dwrap·21()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1306 +0x47
       
      Previous read at 0x00c0002d47b8 by goroutine 306:
        github.com/couchbase/sync_gateway/db.(*BackgroundManager).markStart.func1()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/background_mgr.go:214 +0x84
       
      Goroutine 37 (running) created at:
        testing.(*T).Run()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1306 +0x726
        testing.runTests.func1()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1598 +0x99
        testing.tRunner()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1259 +0x22f
        testing.runTests()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1596 +0x7ca
        testing.(*M).Run()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1504 +0x9d1
        github.com/couchbase/sync_gateway/rest.TestMain()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/main_test.go:36 +0x3ea
        main.main()
            _testmain.go:1325 +0x264
       
      Goroutine 306 (finished) created at:
        github.com/couchbase/sync_gateway/db.(*BackgroundManager).markStart()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/background_mgr.go:204 +0x707
        github.com/couchbase/sync_gateway/db.(*BackgroundManager).Start()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/db/background_mgr.go:96 +0x66
        github.com/couchbase/sync_gateway/rest.(*handler).handleCompact()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/api.go:159 +0xb2b
        github.com/couchbase/sync_gateway/rest.(*handler).invoke()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/handler.go:454 +0x38b9
        github.com/couchbase/sync_gateway/rest.makeHandler.func1()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/handler.go:126 +0xfb
        net/http.HandlerFunc.ServeHTTP()
            /Users/isaaclambat/sdk/go1.17.11/src/net/http/server.go:2047 +0x4d
        github.com/gorilla/mux.(*Router).ServeHTTP()
            /Users/isaaclambat/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210 +0x366
        github.com/couchbase/sync_gateway/rest.wrapRouter.func1()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/routing.go:346 +0x8f0
        net/http.HandlerFunc.ServeHTTP()
            /Users/isaaclambat/sdk/go1.17.11/src/net/http/server.go:2047 +0x4d
        github.com/couchbase/sync_gateway/rest.(*RestTester).SendAdminRequest()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/utilities_testing.go:605 +0x451
        github.com/couchbase/sync_gateway/rest.TestAttachmentCompactionAPI()
            /Users/isaaclambat/dev/sync-gateway/master/sync_gateway/rest/attachment_compaction_api_test.go:38 +0x48f
        testing.tRunner()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1259 +0x22f
        testing.(*T).Run·dwrap·21()
            /Users/isaaclambat/sdk/go1.17.11/src/testing/testing.go:1306 +0x47
      ==================
      

      Attachments

        Activity

          People

            isaac.lambat Isaac Lambat
            adamf Adam Fraser
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty