Simple rebalance puts indexer into a bad state

After this test runs

./testrunner -i VMs.ini -p doc-per-day=6,primary_indx_type=GSI,skip_cleanup=True -t tuqquery.tuq_concurrent.ConcurrentTests.test_concurrent_queries,test_to_run=test_group_by_having,ops=rebalance,GROUP=CONCURRENT

Index creation fails with this message

CREATE PRIMARY INDEX ON `default` USING gsi

[

{ "code": 5000, "msg": "GSI CreatePrimaryIndex() - cause: Fails to create index. There is no available index service that can process this request at this time. Index Service can be in bootstrap, recovery, or non-reachable. Please retry the operation at a later time.", "query": "CREATE PRIMARY INDEX ON `default` USING gsi" }

]

The test essentially runs this query (which is admittedly a weird query) then rebalances in a node while this query is running concurrently (4 threads):

from default WHERE join_mo>7 GROUP BY tasks_points.task1 HAVING COUNT(tasks_points.task1) > 0 SELECT tasks_points.task1 AS task ORDER BY tasks_points.task1

After the test we cannot run any queries because it says there is no index available (there is a primary index built).

select * from default

[

{ "code": 4000, "msg": "No index available on keyspace default that matches your query. Use CREATE INDEX or CREATE PRIMARY INDEX to create an index, or check that your expected index is online.", "query": "select * from default" }

]

Logs attached, backup attached

UI logs show that a lot of services crash after the rebalance starts

Steps to reproduce :

1. Set up a 1 node cluster with kv-index-query services
2. Load travel-sample
3. Rebalance in a data node
4. Try to create a primary index on travel-sample (fails because it says there is no index service available)
5. Try to run a simple query (select * from `travel-sample`) (fails because it thinks there are no indexes available, however there is even a primary index on travel-sample)