Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57968

[Backport MB-57907 to 7.2.1] [System Test OnPrem] Rebalance failure - ServiceAPI.PrepareTopologyChange

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes

    Description

      Not 100% sure on the exact steps to reproduce but this did occur a few times, so presumably should work.

      Create a cluster with KV + GSI + Query nodes.
      Add a new indexer node and rebalance in.
      Go to the newly added node and kill indexer. This kills the rebalance
      Now add the node again and rebalance in again. This leads to rebalance failure with this error.

      {agent_died,<32687.4448.254>,
      {linked_process_died,<32687.6787.254>,
      {'ns_1@172.23.97.108',
      {timeout,
      {gen_server,call,
      [<32687.5557.254>,
      {call,"ServiceAPI.PrepareTopologyChange",
      #Fun<json_rpc_connection.0.69248800>,
      #{timeout => 60000}},
      60000]}}}}}}.
      

      Could be due to the recent changes for MB-57057. I've marked this as a test blocker because the system test does a lot of indexer kills.
      cbcollect ->

      /https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.105.122.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.171.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.176.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.30.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.198.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.230.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.245.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.100.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.108.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.109.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.66.zip
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.67.zip
      

      Attachments

        Issue Links

          Activity

            People

              deepkaran.salooja Deepkaran Salooja
              amit.kulkarni Amit Kulkarni
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty