[System Test] Rebalance failure - Missing real defn id when rebalancing partitioned index

Rebalance failure message -

[user:error,2024-02-11T06:59:43.600-08:00,ns_1@172.23.97.67:<0.50.317>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {service_rebalance_failed,index,

                              {worker_died,

                               {'EXIT',<0.15537.1097>,

                                {task_failed,rebalance,

                                 {service_error,

                                  <<"Missing real defn id when rebalancing partitioned index">>}}}}}.

Rebalance Operation Id = 72cda1692737f492f738e43ac7010be1

This was seen around iteration 33. I had to manually rerun eagle-eye because of a failed run earlier, so the exact iteration number may not be accurate.

Cbcollect logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.67.zip

One caveat is that the cluster has fewer index nodes and thus it might be undersized. At some point of the test, the cluster has ended up with 3 index nodes as opposed to 5 or 6. I'n unsure if that has anything to do with the rebalance failure.

cbcollect ->

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.67.zip

Older logs (n-1)

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.67.zip

Older logs (n-2)

Cbcollect logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.105.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.67.zip

Older logs (n-3)

Cbcollect logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.106.30.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.198.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.230.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.108.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.66.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.67.zip