Tommie McAfee added a comment - 26/Sep/18 9:04 AM In fact we should probably let updateMembers add every node we get from ClusterInfo.Nodes to c.members since they would have all been members prior to operator crash.  The operator would just continue to reconcile members based on their state.  

Tommie McAfee added a comment - 26/Sep/18 8:58 AM Looks like the node was auto-failedover between the time the operator was killed and begun it's next reconcile.  (30s) [user:info, 2018 - 09 -25T23: 55 : 49 .926Z,ns_1 @test -couchbase-54gn7- 0000 .test-couchbase-54gn7. default .svc:< 0.715 . 0 >:auto_failover:log_failover_success: 561 ]Node ( 'ns_1@test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc' ) was automatically failed over. Reason: All monitors report node is unhealthy.   Since member is not yet rebalance out it is in FailedNode state, which leads to issue here since we don't recognize nodes in that state as valid members. func (c *Cluster) updateMembers(known couchbaseutil.MemberSet) error { ... members := couchbaseutil.MemberSet{}   members.Append(status.ActiveNodes)   members.Append(status.PendingAddNodes)   members.Append(status.FailedAddNodes)   members.Append(status.DownNodes)   And when node is omitted from cluster members the operator identifies it as unmanaged, deletes it, and creates a new node. time= "2018-09-25T23:55:54Z" level=info msg= "unmanaged nodes: test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc:8091" cluster-name=test-couchbase-54gn7 module=cluster   So one way to fix this is to add FailedNodes when we call updateMembers.  I can't think of any issues with that.  

Simon Murray added a comment - 26/Sep/18 2:58 AM Recreate locally: time="2018-09-26T09:40:01Z" level=info msg="Node status:" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="┌─────────────────┬──────────────┬────────────────┐" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ Server │ Class │ Status │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="├─────────────────┼──────────────┼────────────────┤" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0000 │ all_services │ managed+active │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0001 │ all_services │ managed+active │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0002 │ all_services │ managed+down │" cluster-name=cb-example module=cluster time="2018-09-26T09:40:01Z" level=info msg="└─────────────────┴──────────────┴────────────────┘" cluster-name=cb-example module=cluster All running pods and PVCs are correctly aggregated (in updateMembers()) and the state is correct.  There is an entry for: persistentvolumeclaim/pvc-couchbase-test-couchbase-54gn7-0001-00-default/pvc-couchbase-test-couchbase-54gn7-0001-00-default.yaml In the cbopinfo collection and the labels look correct. Tommie McAfee anything jump out at you?  Seems test-couchbase-54gn7-0001 isn't getting picked up somehow on restart, and I'm not seeing this behaviour in minikube