Details
-
Bug
-
Resolution: Not a Bug
-
Critical
-
Columnar 1.0.0
-
1.0.0-2203-columnar
-
Untriaged
-
Linux x86_64
-
0
-
Unknown
-
Analytics Sprint 46
Description
Issue
System state is marked as corrupted for multiple nodes in a Columnar instance after triggering a scaling operation for that particular cluster.
For example - consider node 010, System state is marked as healthy before scaling operation.
2024-07-10T14:07:40.867+00:00 INFO CBAS.bootstrap.NCApplication [main] System state: HEALTHY |
2024-07-10T14:07:40.867+00:00 INFO CBAS.bootstrap.NCApplication [main] Node ID: 43186f0aaf75b714f25d14d7b125168a |
Scaling operation is triggered reducing the number of nodes in a Columnar instance from 16 -> 8.
2024-07-10T14:11:58.957Z, ns_orchestrator:0:info:message(ns_1@svc-da-node-003.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com) - Starting rebalance, KeepNodes = ['ns_1@svc-da-node-002.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-005.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-008.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-009.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-010.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-012.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-014.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-016.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com'], EjectNodes = ['ns_1@svc-da-node-001.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-003.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-004.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-006.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-007.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-011.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-013.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com', |
'ns_1@svc-da-node-015.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 6e2b484b00870dd3472762b3102289d4 |
2024-07-10T14:12:11.093Z, ns_orchestrator:0:info:message(ns_1@svc-da-node-003.dlofr1hviuzgwzaf.sandbox.nonprod-project-avengers.com) - Rebalance completed successfully. |
Rebalance Operation Id = 6e2b484b00870dd3472762b3102289d4
|
System state is marked as corrupted when Analytics driver is restarted after completion of scaling operation.
2024-07-10T14:12:14.812+00:00 INFO CBAS.bootstrap.NCApplication [main] System state: CORRUPTED |
2024-07-10T14:12:14.812+00:00 INFO CBAS.bootstrap.NCApplication [main] Node ID: 43186f0aaf75b714f25d14d7b125168a |
Note
This issue is seen for Columnar instance 2203 which contains the fix for MB-62615, hence filing a new ticket.