Sync Gateway not handling graceful failover correctly

Originally filed at https://github.com/couchbase/sync_gateway/issues/4223 by GitHub user olivierboudet

Sync Gateway version

docker image 2.1.2-community

Operating system

Linux Ubuntu 18.04

Config file

{

"adminInterface":"0.0.0.0:4985",

"interface":"0.0.0.0:4984",

"databases": {

"structure": {

"unsupported": {

"user_views": {

"enabled": true

}

},

"server": "http://couchbase-service:8091",

"allow_empty_password": false,

"bucket": "structure",

"username": "syncgateway",

"password": "syncgateway",

"use_views": true,

"sync": "function(doc, oldDoc) {channel(doc.model+'_'+doc.identifiantSite)};",

"users": {

"GUEST": {

"disabled": true,

"admin_channels": [

"*"

],

"all_channels": [

"*"

]

}

}

}

}

}

couchbase-service is a k8s service with these endpoints :

kprod describe endpoints couchbase-service

Name: couchbase-service

Namespace: default

Labels: app=couchbase-worker-pod

chart=couchbase-1.0.0

heritage=Tiller

release=couchbase

Annotations: <none>

Subsets:

Addresses: 10.32.2.189,10.32.7.13,10.32.8.17

NotReadyAddresses: <none>

Ports:

Name Port Protocol

---- ---- --------

http-views 8092 TCP

http 8091 TCP

Events: <none>

Log output

https://gist.github.com/olivierboudet/926f2407d42d42a6e8f6e86f7c3be12b

Expected behavior

When doing a graceful failover of a node, data must be available for user via the sync gateway.

Actual behavior

In a 3 nodes cluster with one instance of SG, when doing a graceful failover of a node, user can not fetch data from the sync gateway. In the logs I can see multiple occurence of the line :

"2019-08-30T09:50:50.397Z [WRN] MultiChangesFeed got error reading changes feed \"STRUCTURE_22RW7\": unauthorized - password required -- db.(*Database).SimpleMultiChangesFeed.func1() at changes.go:493"

This error disappear as soon as the failover is done, and the user can fetch data again.