Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.1.0
Description
Testcase: TestPersistentVolumeKillPodAndOperator
Scenario:
- Couchbase cluster of size 4 is created with default persistent volume mount
- Operator and pod 0001 is killed in parallel
Observation:
Once operator respawned, it is creating a new pod 0004 to replace the killed pod without attempting delta recovery of pod 0001
Cluster events:
Event schema validation failed:
|
NewMemberAdded | New member test-couchbase-54gn7-0000 added to cluster
|
NewMemberAdded | New member test-couchbase-54gn7-0001 added to cluster
|
NewMemberAdded | New member test-couchbase-54gn7-0002 added to cluster
|
NewMemberAdded | New member test-couchbase-54gn7-0003 added to cluster
|
RebalanceStarted | A rebalance has been started to balance data across the cluster
|
RebalanceCompleted | A rebalance has completed
|
BucketCreated | A new bucket `PVBucket` was created
|
NewMemberAdded | New member test-couchbase-54gn7-0004 added to cluster | <== no anyof members matched
|
RebalanceStarted | A rebalance has been started to balance data across the cluster
|
MemberRemoved | Existing member test-couchbase-54gn7-0001 removed from the cluster
|
RebalanceCompleted | A rebalance has completed
|
Attachments
Activity
Looks like the node was auto-failedover between the time the operator was killed and begun it's next reconcile. (30s)
[user:info,2018-09-25T23:55:49.926Z,ns_1@test-couchbase-54gn7-0000.test-couchbase-54gn7.default.svc:<0.715.0>:auto_failover:log_failover_success:561]Node ('ns_1@test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc') was automatically failed over. Reason: All monitors report node is unhealthy. |
Since member is not yet rebalance out it is in FailedNode state, which leads to issue here since we don't recognize nodes in that state as valid members.
func (c *Cluster) updateMembers(known couchbaseutil.MemberSet) error {
|
...
|
members := couchbaseutil.MemberSet{}
|
members.Append(status.ActiveNodes)
|
members.Append(status.PendingAddNodes)
|
members.Append(status.FailedAddNodes)
|
members.Append(status.DownNodes)
|
And when node is omitted from cluster members the operator identifies it as unmanaged, deletes it, and creates a new node.
time="2018-09-25T23:55:54Z" level=info msg="unmanaged nodes: test-couchbase-54gn7-0001.test-couchbase-54gn7.default.svc:8091" cluster-name=test-couchbase-54gn7 module=cluster |
So one way to fix this is to add FailedNodes when we call updateMembers. I can't think of any issues with that.
In fact we should probably let updateMembers add every node we get from ClusterInfo.Nodes to c.members since they would have all been members prior to operator crash. The operator would just continue to reconcile members based on their state.
Recreate locally:
time="2018-09-26T09:40:01Z" level=info msg="Node status:" cluster-name=cb-example module=clustertime="2018-09-26T09:40:01Z" level=info msg="┌─────────────────┬──────────────┬────────────────┐" cluster-name=cb-example module=clustertime="2018-09-26T09:40:01Z" level=info msg="│ Server │ Class │ Status │" cluster-name=cb-example module=clustertime="2018-09-26T09:40:01Z" level=info msg="├─────────────────┼──────────────┼────────────────┤" cluster-name=cb-example module=clustertime="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0000 │ all_services │ managed+active │" cluster-name=cb-example module=clustertime="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0001 │ all_services │ managed+active │" cluster-name=cb-example module=clustertime="2018-09-26T09:40:01Z" level=info msg="│ cb-example-0002 │ all_services │ managed+down │" cluster-name=cb-example module=clustertime="2018-09-26T09:40:01Z" level=info msg="└─────────────────┴──────────────┴────────────────┘" cluster-name=cb-example module=clusterAll running pods and PVCs are correctly aggregated (in updateMembers()) and the state is correct. There is an entry for:
persistentvolumeclaim/pvc-couchbase-test-couchbase-54gn7-0001-00-default/pvc-couchbase-test-couchbase-54gn7-0001-00-default.yamlIn the cbopinfo collection and the labels look correct.
Tommie McAfee anything jump out at you? Seems test-couchbase-54gn7-0001 isn't getting picked up somehow on restart, and I'm not seeing this behaviour in minikube