Loading...

Details

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: Morpheus, 7.6.4
Affects Version/s: 7.6.2
Component/s: secondary-index
Labels:
- AMI
- ami
- capella-test
- file-based-rebalance
- upgrade

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-006.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-007.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-008.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-009.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-010.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip
---------------------------------------------------------------------------
https://app.datadoghq.com/logs?query=@clusterId:15673c72-9845-458d-a876-48e79c03e022%20status:error%20@msg:%22the%20cluster%20is%20not%20balanced%20-%20can%20not%20perform%20operations%20on%20cluster%20nodes%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host,service&event=AgAAAZCiZjl1byWXwgAAAAAAAAAYAAAAAEFaQ2laa1Z6QUFEZkxYZmsySVh4SEFBRgAAACQAAAAAMDE5MGEyNjYtNGU0Ni00NDRhLWI2MTItOTJjMzU5YTg3Y2Yz&fromUser=true&messageDisplay=inline&storage=hot&stream_sort=desc&viz=stream&from_ts=1720538675767&to_ts=1720711475767&live=true
https://app.datadoghq.com/logs?query=@clusterId:15673c72-9845-458d-a876-48e79c03e022%20status:error%20@msg:%22attempting%20to%20obtain%20configs%20for%20node%20during%20the%20registration%20process%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host,service&event=AgAAAZCcyihZG2XY9wAAAAAAAAAYAAAAAEFaQ2N5akI4QUFBSVlzZnpHRlV1VlFBdgAAACQAAAAAMDE5MDlkNWYtN2ZlZS00N2MyLWJiZGYtMTI1MDAzMzk0ZDNk&fromUser=true&messageDisplay=inline&storage=hot&stream_sort=desc&viz=stream&from_ts=1720538675767&to_ts=1720711475767&live=true

Show
https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-006.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-007.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-008.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-009.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip https://cb-engineering.s3.amazonaws.com/aman/collectinfo-2024-07-11T090354-ns_1%40svc-dqisea-node-010.2utcxfk1o3j135bd.sandbox.nonprod-project-avengers.com.zip --------------------------------------------------------------------------- https://app.datadoghq.com/logs?query=@clusterId:15673c72-9845-458d-a876-48e79c03e022%20status:error%20@msg:%22the%20cluster%20is%20not%20balanced%20-%20can%20not%20perform%20operations%20on%20cluster%20nodes%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host,service&event=AgAAAZCiZjl1byWXwgAAAAAAAAAYAAAAAEFaQ2laa1Z6QUFEZkxYZmsySVh4SEFBRgAAACQAAAAAMDE5MGEyNjYtNGU0Ni00NDRhLWI2MTItOTJjMzU5YTg3Y2Yz&fromUser=true&messageDisplay=inline&storage=hot&stream_sort=desc&viz=stream&from_ts=1720538675767&to_ts=1720711475767&live=true https://app.datadoghq.com/logs?query=@clusterId:15673c72-9845-458d-a876-48e79c03e022%20status:error%20@msg:%22attempting%20to%20obtain%20configs%20for%20node%20during%20the%20registration%20process%22&agg_m=count&agg_m_source=base&agg_t=count&cols=host,service&event=AgAAAZCcyihZG2XY9wAAAAAAAAAYAAAAAEFaQ2N5akI4QUFBSVlzZnpHRlV1VlFBdgAAACQAAAAAMDE5MDlkNWYtN2ZlZS00N2MyLWJiZGYtMTI1MDAzMzk0ZDNk&fromUser=true&messageDisplay=inline&storage=hot&stream_sort=desc&viz=stream&from_ts=1720538675767&to_ts=1720711475767&live=true
Story Points:
0
Is this a Regression?:
Unknown

Description

While running an upgrade test on the AMI - 7.6.2-3721, the Capella cluster was stuck at a state of `scaling` after the upgrade cluster got completed and the cluster was scaling in from 5 to 3 nodes.
(Screenshot attached below)

1 - Bucket Specs :
- DISK : 10,641 MiB
- RAM quota : 3,228 MiB / 5,500 MiB
2. All services are colocated on all nodes, max index size: 80mb
3 - CSP : azure, Region : East US (Virginia), AMI :
- (deployed at) : couchbase-cloud-server-7.2.5-v7596.0.32
- (upgraded to) : couchbase-cloud-server-7.6.2-v3721.0.35
4 - During the action to scale-in from 5 to 3 nodes, gsi rebalance got stuck at 98.5% .
5- server logs + dd logs (attached below)
6 - cluster alive and can be found here

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

Screenshot 2024-07-11 at 8.52.16 PM.png
87 kB
11/Jul/24 8:22 AM

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[Capella][File based rebalance] Azure Cluster stuck at Scaling state after upgrade to 7.6.2

Details

Description

Attachments

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty