Details
-
Bug
-
Resolution: Fixed
-
Minor
-
Columnar 1.0.0
-
Untriaged
-
0
-
Unknown
-
Analytics Sprint 49
Description
is isBalanced check takes place while cluster is recovering, the response isn't (sent or handled) properly.
2024-08-19T22:57:11.861+00:00 INFO CBAS.servlet.RebalanceServlet [HttpExecutor(port:9111)-9] +post request: {"nodes":[{"nodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","priority":281474976776448,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":0,"host":"svc-da-node-001.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":0,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}},{"nodeId":"00e8c0a07f2a167fd9505b0a135d806c","priority":281474976776192,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":1,"host":"svc-da-node-002.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":16,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}},{"nodeId":"f066f1c77e314045e23015c5831a0263","priority":281474976776192,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":2,"host":"svc-da-node-003.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":32,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}},{"nodeId":"6cfc42f243b1fd766abff7b13ab6e752","priority":281474976776192,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":3,"host":"svc-da-node-004.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":48,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}}],"id":"3ca6a1a696212ac32be2bd76d4706cb7","type":"topology-change-failover","ccNodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","metadataNodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","metadataPartition":-1,"rev":5,"configVersion":1,"balanceState":"unknown","keepNodesUpdated":true,"keepNodes":["a01a5c04a60206bfdffe9e7cf4b3a43f","00e8c0a07f2a167fd9505b0a135d806c","6cfc42f243b1fd766abff7b13ab6e752"],"inPlaceNumReplicas":0,"balanceStateClusterCompat":458758,"orchestratorNodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","serverGroupsVersion":12024846,"ejectNodes":[],"columnarCompatLevel":"1.0.1.0"}
|
|
2024-08-19T22:57:11.868+00:00 INFO CBAS.server.AbstractServlet [HttpExecutor(port:9111)-9] sendError: status=503 Service Unavailable, message=Service Unavailable: cannot verify balance state due to cluster state RECOVERING; try again later.
|
2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas Error unmarshalling isBalanced response: invalid character 'S' looking for beginning of value
|
2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas request to resolveBalance failed (invalid character 'S' looking for beginning of value); sleeping 15s before attempting again...
|
More precisely, the last part of the above log:
2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas Error unmarshalling isBalanced response: invalid character 'S' looking for beginning of value2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas request to resolveBalance failed (invalid character 'S' looking for beginning of value); sleeping 15s before attempting again...
|