Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-620

Concurrent attempts to create indexes trigger SG restart

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: SyncGateway
    • Security Level: Public
    • Labels:
      None
    • Required Mobile Fields:
      Hide
      Mandatory:
       - CBL / SG Version:
         - SG Config:
       - Steps to Reproduce:
       - Actual Result:
       - Expected Result:
       - Logs :
            SGW LOGS: sgcollect info
            CBL LOGS:
            Logcat LOGS: for Android tickets
       - Github link for the code:
       - Jenkins job failure link:
       - Pytest Command
       - What is the last build this test passed:
      Show
      Mandatory:  - CBL / SG Version:    - SG Config:  - Steps to Reproduce:  - Actual Result:  - Expected Result:  - Logs :       SGW LOGS: sgcollect info       CBL LOGS:       Logcat LOGS: for Android tickets  - Github link for the code:  - Jenkins job failure link:  - Pytest Command  - What is the last build this test passed:
    • Sprint:
      CBG Sprint 37
    • Story Points:
      2

      Description

      When multiple Sync Gateway nodes concurrently attempt to initialize indexes, the Sync Gateway node can fail with an error like:

      2019-12-06T15:11:05.136-08:00 [WRN] Error when querying index using statement: [SELECT meta(`bucket-1`).xattrs._sync.access.foo as val FROM `bucket-1` WHERE ANY op in OBJECT_PAIRS(meta(`bucket-1`).xattrs._sync.access) SATISFIES op.name = 'foo' end LIMIT 1] parameters: [<nil>] error:[5000] Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       - cause: Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       -- base.(*CouchbaseBucketGoCB).Query() at bucket_n1ql.go:76
      2019-12-06T15:11:05.136-08:00 [WRN] Query error for statement [SELECT meta(`$_bucket`).xattrs._sync.access.foo as val FROM `$_bucket` WHERE ANY op in OBJECT_PAIRS(meta(`$_bucket`).xattrs._sync.access) SATISFIES op.name = 'foo' end LIMIT 1], err:[5000] Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       - cause: Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       -- db.waitForIndexes.func1() at indexes.go:335
      2019-12-06T15:11:05.145-08:00 [ERR] Error opening database db: [5000] Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       - cause: Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       
      github.com/couchbase/sync_gateway/base.(*CouchbaseBucketGoCB).Query
              /home/couchbase/jenkins/workspace/sgw-toy-build/2.7.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/bucket_n1ql.go:77
      github.com/couchbase/sync_gateway/db.waitForIndex
              /home/couchbase/jenkins/workspace/sgw-toy-build/2.7.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/indexes.go:358
      github.com/couchbase/sync_gateway/db.waitForIndexes.func1
              /home/couchbase/jenkins/workspace/sgw-toy-build/2.7.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/indexes.go:333
      runtime.goexit
              /usr/local/go/1.13.4/go/src/runtime/asm_amd64.s:1357 -- rest.RunServer() at config.go:1028
      

      We should detect and retry/wait based on this type of error, instead of exiting.

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.7.0-160 contains sync_gateway commit 003bc5f with commit message:
          CBG-620 backport to 2.7.0 (#4396)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.7.0-160 contains sync_gateway commit 003bc5f with commit message: CBG-620 backport to 2.7.0 (#4396)
          Hide
          build-team Couchbase Build Team added a comment -

          Build sync_gateway-2.8.0-12 contains sync_gateway commit 03bf2ab with commit message:
          CBG-620 - Retry on "Build Already In Progress" errors (#4394)

          Show
          build-team Couchbase Build Team added a comment - Build sync_gateway-2.8.0-12 contains sync_gateway commit 03bf2ab with commit message: CBG-620 - Retry on "Build Already In Progress" errors (#4394)
          Hide
          adamf Adam Fraser added a comment -

          Moving this to Mercury as a potential regression, as it's showing up much more frequently in Mercury. If it's found to be a non-regression after further analysis, can be moved back to Hydrogen.

          Show
          adamf Adam Fraser added a comment - Moving this to Mercury as a potential regression, as it's showing up much more frequently in Mercury. If it's found to be a non-regression after further analysis, can be moved back to Hydrogen.

            People

            Assignee:
            ben.brooks Ben Brooks
            Reporter:
            adamf Adam Fraser
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty