Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-500

Sync Gateway not handling graceful failover correctly

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.2
    • Fix Version/s: 2.7.0
    • Component/s: SyncGateway
    • Security Level: Public
    • Labels:
      None
    • Sprint:
      CBG Sprint 30, CBG Sprint 32
    • Story Points:
      1

      Description

      Originally filed at https://github.com/couchbase/sync_gateway/issues/4223 by GitHub user olivierboudet

      Sync Gateway version

      docker image 2.1.2-community

      Operating system

      Linux Ubuntu 18.04

      Config file

      {
      	"adminInterface":"0.0.0.0:4985",
      	"interface":"0.0.0.0:4984",
      	"databases": {
      		"structure": {
      			"unsupported": {
      				"user_views": {
      				  "enabled": true
      				}
      			},
      			"server": "http://couchbase-service:8091",
      			"allow_empty_password": false,
      			"bucket": "structure",
      			"username": "syncgateway",
      			"password": "syncgateway",
      			"use_views": true,
      			"sync": "function(doc, oldDoc) {channel(doc.model+'_'+doc.identifiantSite)};",
      			"users": {
      				"GUEST": {
      					"disabled": true,
      					"admin_channels": [
      						"*"
      					],
      					"all_channels": [
      						"*"
      					]
      				}
      			}
      		}
      	}
      }
      

      couchbase-service is a k8s service with these endpoints :

      kprod describe endpoints couchbase-service
      Name:         couchbase-service
      Namespace:    default
      Labels:       app=couchbase-worker-pod
                    chart=couchbase-1.0.0
                    heritage=Tiller
                    release=couchbase
      Annotations:  <none>
      Subsets:
        Addresses:          10.32.2.189,10.32.7.13,10.32.8.17
        NotReadyAddresses:  <none>
        Ports:
          Name        Port  Protocol
          ----        ----  --------
          http-views  8092  TCP
          http        8091  TCP
       
      Events:  <none>
      

      Log output

      https://gist.github.com/olivierboudet/926f2407d42d42a6e8f6e86f7c3be12b

      Expected behavior

      When doing a graceful failover of a node, data must be available for user via the sync gateway.

      Actual behavior

      In a 3 nodes cluster with one instance of SG, when doing a graceful failover of a node, user can not fetch data from the sync gateway. In the logs I can see multiple occurence of the line :

      "2019-08-30T09:50:50.397Z [WRN] MultiChangesFeed got error reading changes feed \"STRUCTURE_22RW7\": unauthorized - password required -- db.(*Database).SimpleMultiChangesFeed.func1() at changes.go:493"
      

      This error disappear as soon as the failover is done, and the user can fetch data again.

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          adamf Adam Fraser added a comment -

          Related issue https://issues.couchbase.com/browse/GOCBC-591 has been fixed - need to uptake in the SG manifest.

          Show
          adamf Adam Fraser added a comment - Related issue https://issues.couchbase.com/browse/GOCBC-591 has been fixed - need to uptake in the SG manifest.
          Hide
          adamf Adam Fraser added a comment -

          Addressed with gocb uptake in CBG-488 / https://github.com/couchbase/sync_gateway/pull/4247.

           

           

          Show
          adamf Adam Fraser added a comment - Addressed with gocb uptake in CBG-488 /  https://github.com/couchbase/sync_gateway/pull/4247 .    

            People

            Assignee:
            adamf Adam Fraser
            Reporter:
            ben.brooks Ben Brooks
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty