Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-620

Concurrent attempts to create indexes trigger SG restart

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: SyncGateway
    • Security Level: Public
    • Labels:
      None
    • Required Mobile Fields:
      Hide
      Mandatory:
       - CBL / SG Version:
         - SG Config:
       - Steps to Reproduce:
       - Actual Result:
       - Expected Result:
       - Logs :
            SGW LOGS: sgcollect info
            CBL LOGS:
            Logcat LOGS: for Android tickets
       - Github link for the code:
       - Jenkins job failure link:
       - Pytest Command
       - What is the last build this test passed:
      Show
      Mandatory:  - CBL / SG Version:    - SG Config:  - Steps to Reproduce:  - Actual Result:  - Expected Result:  - Logs :       SGW LOGS: sgcollect info       CBL LOGS:       Logcat LOGS: for Android tickets  - Github link for the code:  - Jenkins job failure link:  - Pytest Command  - What is the last build this test passed:
    • Sprint:
      CBG Sprint 37
    • Story Points:
      2

      Description

      When multiple Sync Gateway nodes concurrently attempt to initialize indexes, the Sync Gateway node can fail with an error like:

      2019-12-06T15:11:05.136-08:00 [WRN] Error when querying index using statement: [SELECT meta(`bucket-1`).xattrs._sync.access.foo as val FROM `bucket-1` WHERE ANY op in OBJECT_PAIRS(meta(`bucket-1`).xattrs._sync.access) SATISFIES op.name = 'foo' end LIMIT 1] parameters: [<nil>] error:[5000] Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       - cause: Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       -- base.(*CouchbaseBucketGoCB).Query() at bucket_n1ql.go:76
      2019-12-06T15:11:05.136-08:00 [WRN] Query error for statement [SELECT meta(`$_bucket`).xattrs._sync.access.foo as val FROM `$_bucket` WHERE ANY op in OBJECT_PAIRS(meta(`$_bucket`).xattrs._sync.access) SATISFIES op.name = 'foo' end LIMIT 1], err:[5000] Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       - cause: Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       -- db.waitForIndexes.func1() at indexes.go:335
      2019-12-06T15:11:05.145-08:00 [ERR] Error opening database db: [5000] Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       - cause: Index sg_access_x1 will retry building in the background for reason: Build Already In Progress. Bucket bucket-1.
       
      github.com/couchbase/sync_gateway/base.(*CouchbaseBucketGoCB).Query
              /home/couchbase/jenkins/workspace/sgw-toy-build/2.7.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/bucket_n1ql.go:77
      github.com/couchbase/sync_gateway/db.waitForIndex
              /home/couchbase/jenkins/workspace/sgw-toy-build/2.7.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/indexes.go:358
      github.com/couchbase/sync_gateway/db.waitForIndexes.func1
              /home/couchbase/jenkins/workspace/sgw-toy-build/2.7.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/db/indexes.go:333
      runtime.goexit
              /usr/local/go/1.13.4/go/src/runtime/asm_amd64.s:1357 -- rest.RunServer() at config.go:1028
      

      We should detect and retry/wait based on this type of error, instead of exiting.

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          adamf Adam Fraser created issue -
          adamf Adam Fraser made changes -
          Field Original Value New Value
          Fix Version/s Hydrogen [ 16179 ]
          adamf Adam Fraser made changes -
          Fix Version/s Hydrogen [ 16179 ]
          Fix Version/s Mercury [ 15421 ]
          adamf Adam Fraser made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          adamf Adam Fraser made changes -
          Assignee The One [ the one ] Adam Fraser [ adamf ]
          adamf Adam Fraser made changes -
          Story Points 2
          adamf Adam Fraser made changes -
          Sprint CBG Sprint 37 [ 943 ]
          adamf Adam Fraser made changes -
          Rank Ranked higher
          adamf Adam Fraser made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          ben.brooks Ben Brooks made changes -
          Assignee Adam Fraser [ adamf ] Ben Brooks [ ben.brooks ]
          Automated transition triggered when Ben Brooks created pull request #4394 in GitHub -
          Status In Progress [ 3 ] In Review [ 10107 ]
          Automated transition triggered when Adam Fraser merged pull request #4394 in GitHub -
          Resolution Fixed [ 1 ]
          Status In Review [ 10107 ] Resolved [ 5 ]
          adamf Adam Fraser made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            Assignee:
            ben.brooks Ben Brooks
            Reporter:
            adamf Adam Fraser
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty