Uploaded image for project: 'Couchbase Gateway'
  1. Couchbase Gateway
  2. CBG-500

Sync Gateway not handling graceful failover correctly

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.1.2
    • 2.7.0
    • SyncGateway
    • Security Level: Public
    • None
    • CBG Sprint 30, CBG Sprint 32
    • 1

    Description

      Originally filed at https://github.com/couchbase/sync_gateway/issues/4223 by GitHub user olivierboudet

      Sync Gateway version

      docker image 2.1.2-community

      Operating system

      Linux Ubuntu 18.04

      Config file

      {
      	"adminInterface":"0.0.0.0:4985",
      	"interface":"0.0.0.0:4984",
      	"databases": {
      		"structure": {
      			"unsupported": {
      				"user_views": {
      				  "enabled": true
      				}
      			},
      			"server": "http://couchbase-service:8091",
      			"allow_empty_password": false,
      			"bucket": "structure",
      			"username": "syncgateway",
      			"password": "syncgateway",
      			"use_views": true,
      			"sync": "function(doc, oldDoc) {channel(doc.model+'_'+doc.identifiantSite)};",
      			"users": {
      				"GUEST": {
      					"disabled": true,
      					"admin_channels": [
      						"*"
      					],
      					"all_channels": [
      						"*"
      					]
      				}
      			}
      		}
      	}
      }
      

      couchbase-service is a k8s service with these endpoints :

      kprod describe endpoints couchbase-service
      Name:         couchbase-service
      Namespace:    default
      Labels:       app=couchbase-worker-pod
                    chart=couchbase-1.0.0
                    heritage=Tiller
                    release=couchbase
      Annotations:  <none>
      Subsets:
        Addresses:          10.32.2.189,10.32.7.13,10.32.8.17
        NotReadyAddresses:  <none>
        Ports:
          Name        Port  Protocol
          ----        ----  --------
          http-views  8092  TCP
          http        8091  TCP
       
      Events:  <none>
      

      Log output

      https://gist.github.com/olivierboudet/926f2407d42d42a6e8f6e86f7c3be12b

      Expected behavior

      When doing a graceful failover of a node, data must be available for user via the sync gateway.

      Actual behavior

      In a 3 nodes cluster with one instance of SG, when doing a graceful failover of a node, user can not fetch data from the sync gateway. In the logs I can see multiple occurence of the line :

      "2019-08-30T09:50:50.397Z [WRN] MultiChangesFeed got error reading changes feed \"STRUCTURE_22RW7\": unauthorized - password required -- db.(*Database).SimpleMultiChangesFeed.func1() at changes.go:493"
      

      This error disappear as soon as the failover is done, and the user can fetch data again.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          adamf Adam Fraser added a comment -

          Related issue https://issues.couchbase.com/browse/GOCBC-591 has been fixed - need to uptake in the SG manifest.

          adamf Adam Fraser added a comment - Related issue https://issues.couchbase.com/browse/GOCBC-591 has been fixed - need to uptake in the SG manifest.
          adamf Adam Fraser added a comment -

          Addressed with gocb uptake in CBG-488 / https://github.com/couchbase/sync_gateway/pull/4247.

           

           

          adamf Adam Fraser added a comment - Addressed with gocb uptake in CBG-488 /  https://github.com/couchbase/sync_gateway/pull/4247 .    

          People

            adamf Adam Fraser
            ben.brooks Ben Brooks
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty