Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-63247

isBalanced response incorrectly parsed in case cluster is not ACTIVE

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown
    • Analytics Sprint 49

    Description

      is isBalanced check takes place while cluster is recovering, the response isn't (sent or handled) properly.

      2024-08-19T22:57:11.861+00:00 INFO CBAS.servlet.RebalanceServlet [HttpExecutor(port:9111)-9] +post request: {"nodes":[{"nodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","priority":281474976776448,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":0,"host":"svc-da-node-001.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":0,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}},{"nodeId":"00e8c0a07f2a167fd9505b0a135d806c","priority":281474976776192,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":1,"host":"svc-da-node-002.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":16,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}},{"nodeId":"f066f1c77e314045e23015c5831a0263","priority":281474976776192,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":2,"host":"svc-da-node-003.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":32,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}},{"nodeId":"6cfc42f243b1fd766abff7b13ab6e752","priority":281474976776192,"opaque":{"cbas-version":"1.0.1-2313","cc-http-port":9111,"controller-id":3,"host":"svc-da-node-004.er65w5qpwe3wffcy.sandbox.nonprod-project-avengers.com","ns-server-port":8091,"num-iodevices":16,"server-group-uri":"/pools/default/serverGroups/c97721434cf63dd8209c90e6741f33e3","starting-partition-id":48,"svc-http-port":8095,"versions":[{"clusterCompat":458758,"columnarCompat":"1.0.1.0","productBuild":"1.0.1-2313"},{"clusterCompat":458758,"columnarCompat":"1.0.0.0","productBuild":"1.0.1-2313"}]}}],"id":"3ca6a1a696212ac32be2bd76d4706cb7","type":"topology-change-failover","ccNodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","metadataNodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","metadataPartition":-1,"rev":5,"configVersion":1,"balanceState":"unknown","keepNodesUpdated":true,"keepNodes":["a01a5c04a60206bfdffe9e7cf4b3a43f","00e8c0a07f2a167fd9505b0a135d806c","6cfc42f243b1fd766abff7b13ab6e752"],"inPlaceNumReplicas":0,"balanceStateClusterCompat":458758,"orchestratorNodeId":"a01a5c04a60206bfdffe9e7cf4b3a43f","serverGroupsVersion":12024846,"ejectNodes":[],"columnarCompatLevel":"1.0.1.0"}
       
      2024-08-19T22:57:11.868+00:00 INFO CBAS.server.AbstractServlet [HttpExecutor(port:9111)-9] sendError: status=503 Service Unavailable, message=Service Unavailable: cannot verify balance state due to cluster state RECOVERING; try again later.
      2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas Error unmarshalling isBalanced response: invalid character 'S' looking for beginning of value
      2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas request to resolveBalance failed (invalid character 'S' looking for beginning of value); sleeping 15s before attempting again...

       

      More precisely, the last part of the above log:

      2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas Error unmarshalling isBalanced response: invalid character 'S' looking for beginning of value2024-08-19T22:57:11.868+00:00 WARN CBAS.cbas request to resolveBalance failed (invalid character 'S' looking for beginning of value); sleeping 15s before attempting again...

      Attachments

        For Gerrit Dashboard: MB-63247
        # Subject Branch Project Status CR V

        Activity

          People

            Hussain.Towaileb Hussain Towaileb
            Hussain.Towaileb Hussain Towaileb
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty