GSI file based rebalance is hung since 12+ hours under new workload of 10k/s, 50-50 Read:Expiry

1. Create a 3KV, 2 GSI, 2 N1QL node capella cluster of compute=c5.2xlarge
2. Create a magma bucket, 2 collections, 100M items in each collection
3. Create 2 indexes per collection.

   Index:

   CREATE INDEX default0_idx_VolumeCollection0_0 ON VolumeCollection0(country, DISTINCT ARRAY `r`.`ratings`.`Check in / front desk` FOR r in `reviews` END,array_count((`public_likes`)),array_count((`reviews`)) DESC,`type`,`phone`,`price`,`email`,`address`,`name`,`url`) PARTITION BY HASH (country) USING GSI WITH { "defer_build": true};

   Query:

   select meta().id from VolumeCollection0 where country is not null and `type` is not null and (any r in reviews satisfies r.ratings.`Check in / front desk` is not null end) limit 100

   Index:

   CREATE INDEX default0_idx_VolumeCollection0_1 ON VolumeCollection0(`free_breakfast`,`type`,`free_parking`,array_count((`public_likes`)),`price`,`country`) PARTITION BY HASH (type) USING GSI WITH { "defer_build": true};

   Query:

   select price, country from VolumeCollection0 where free_breakfast=True AND free_parking=True and price is not null and array_count(public_likes)>=0 and `type`= $type limit 100

   Index:

   CREATE INDEX default0_idx_VolumeCollection1_0 ON VolumeCollection1(`free_breakfast`,`free_parking`,`country`,`city`) PARTITION BY HASH (country) USING GSI WITH { "defer_build": true};

   Query:

   select city,country from VolumeCollection1 where free_breakfast=True and free_parking=True order by country,city limit 100

   Index:

   CREATE INDEX default0_idx_VolumeCollection1_1 ON VolumeCollection1(`country`, `city`,`price`,`name`) PARTITION BY HASH (country, city) USING GSI WITH { "defer_build": true};

   Query:

   WITH city_avg AS (SELECT city, AVG(price) AS avgprice FROM VolumeCollection1 WHERE country = $country GROUP BY city limit 10) SELECT h.name, h.price FROM city_avg JOIN VolumeCollection1 h ON h.city = city_avg.city WHERE h.price < city_avg.avgprice AND h.country=$country limit 100

4. Scale OUT the cluster to 4KV, 3GSI, 3N1QL
5. Scale OUT the cluster to 5KV, 4GSI, 4N1QL
6. Scale OUT the cluster to 6KV, 5GSI, 5N1QL
7. Scale OUT the cluster to 7KV, 6GSI, 6N1QL
8. Scale IN the cluster to 6KV, 5GSI, 5N1QL
9. Scale IN the cluster to 5KV, 4GSI, 4N1QL
10. Scale IN the cluster to 4KV, 3GSI, 3N1QL
11. When it come to last GSI the scale IN rebalance, it is stuck since 12+ hours.

cc: Varun Velamuri, Deepkaran Salooja