Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57907

[System Test OnPrem] Rebalance failure - ServiceAPI.PrepareTopologyChange

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Yes

    Description

      Not 100% sure on the exact steps to reproduce but this did occur a few times, so presumably should work.

      Create a cluster with KV + GSI + Query nodes.
      Add a new indexer node and rebalance in.
      Go to the newly added node and kill indexer. This kills the rebalance
      Now add the node again and rebalance in again. This leads to rebalance failure with this error.

      {agent_died,<32687.4448.254>,
      		 
      {linked_process_died,<32687.6787.254>,
      		 
      {'ns_1@172.23.97.108',
      		 
      {timeout,
      		 
      {gen_server,call,
      		 
      [<32687.5557.254>,
      		 
      {call,"ServiceAPI.PrepareTopologyChange",
      		 
      #Fun<json_rpc_connection.0.69248800>,
      		 
      #{timeout => 60000}},
      		 
      60000]}}}}}}.

      Could be due to the recent changes for MB-57057. I've marked this as a test blocker because the system test does a lot of indexer kills.
      cbcollect ->

      /https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.105.122.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.171.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.176.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.106.30.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.198.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.230.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.96.245.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.100.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.108.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.109.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.66.zip
      		 
      https://cb-engineering.s3.amazonaws.com/Rebalance_Failure_PrepareTopologyChange/collectinfo-2023-07-19T072316-ns_1%40172.23.97.67.zip

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-57968 [Backport MB-57907 to 7.2.1] [System Test OnPrem] Rebalance failure - ServiceAPI.PrepareTopologyChange

          • Test Blocker - QE-related tests can't proceed
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-57873 [System Test]Rebalance failure - inactivity_timeout

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              pavan.pb Pavan PB
              pavan.pb Pavan PB
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty