Details
-
Bug
-
Resolution: Unresolved
-
Major
-
7.6.0
-
CAO Image : couchbase/couchbase-operator:2.7.0-arm64
Couchbase Enterprise Edition 7.6.0-2176
Environment : Kind Kubernetes environment run locally on Mac
-
Untriaged
-
Linux x86_64
-
-
0
-
Unknown
Description
Steps to reproduce
- Created a 3 node cluster on k8s with operator with all services
- On one pod, memcached was killed in a loop. Multiple failovers and rebalance failures occur as expected
- Stopped the memcached kill loop.
- Rebalances beyond this fail in a loop(as triggered by the operator again and again)
Rebalance exited with reason {service_rebalance_failed,eventing,
|
{worker_died,
|
{'EXIT',<0.3701.18>, |
{task_failed,rebalance,
|
{service_error,
|
<<"failed to aggregate rebalance progress from all eventing nodes, err: map[127.0.0.1:8096:Get \"http://127.0.0.1:8096/getAggRebalanceProgress\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)]">>}}}}}. |
Rebalance Operation Id = 73d4b9c58925559a9ce54cb6089d2624
|
During this time tried to query multiple endpoints in the pods
curl -X GET http://127.0.0.1:8096/getAggRebalanceProgress -u Administrator:password |
{
|
"name": "INTERNAL_SERVER_ERROR", |
"code": 59, |
"attributes": [ |
"retry" |
],
|
"description": "Failed to get progress" |
}
|
|
|
curl -X GET http://127.0.0.1:8096/api/v1/status -u Administrator:password |
{
|
"name": "INTERNAL_SERVER_ERROR", |
"code": 59, |
"attributes": [ |
"retry" |
],
|
"description": "Unable to fetch active Eventing nodes, err: Get \"http://cb-example-0001.cb-example.default.svc:8096/uuid\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" |
}
|