Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Columnar 1.0.0
-
1.0.0-2187 4-node cluster (64 GB + 16 vcpus)
-
Untriaged
-
0
-
Unknown
-
Analytics Sprint 45
Description
There are 2 observations -
Rebalance triggered during a scale operation ( 4 nodes to 8 nodes) took almost one hour to complete. The query workload that I have running shouldn't really have that big an impact since the queries are run with a timeout of 10 minutes.
Rebalance Id - 1a6c93c7b58e41e39d81895b6ece9d08
"startTime":"2024-06-30T13:56:24.905Z","completedTime":"2024-06-30T14:59:49.588Z","timeTaken":3804683} |
Post completion of the first rebalance, another rebalance gets triggered. Not sure if this is intentional since the first rebalance was successful.
"rebalanceId":"43e687c67a4b31a90e267114add4cd57" |
"startTime":"2024-06-30T14:59:51.738Z","completedTime":"2024-06-30T14:59:53.010Z","timeTaken":1272 |
cbcollect ->
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-001.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-002.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-003.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-004.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-005.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-006.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-007.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
https://cb-engineering.s3.amazonaws.com/SysTestColumnarJun28/collectinfo-2024-06-30T155358-ns_1%40svc-da-node-008.bks3edqzezfgtl1s.sandbox.nonprod-project-avengers.com.zip
Attachments
For Gerrit Dashboard: MB-62556 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
212234,2 | MB-62556: Add logs around DCP state request | master | cbas-core | Status: MERGED | +2 | +1 |
212364,4 | MB-62556: don't wait for actives to resume to complete rebalance | master | cbas-core | Status: MERGED | +2 | +1 |
212601,4 | MB-62670: revert "MB-62556: don't wait for actives to resume to complete rebalance" | master | cbas-core | Status: MERGED | +2 | +1 |
214779,3 | MB-62556: don't wait for actives to resume to complete rebalance | goldfish | cbas-core | Status: MERGED | +2 | +1 |