Analytics tests with 999 collections failed during the ingestion process

All analytics tests with 999 collections failed during the ingestion process. We don't see this issue in tests with 3 and 99 collections. The tests failed because (1) they didn't get response from analytics/status/ingestion/v2 after 20 attempts, or (2) the response from analytics/status/ingestion/v2 didn't have the right information.

Build: 7.1.0-1250

Job: http://perf.jenkins.couchbase.com/job/oceanus/6884/ 

Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.57.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.5.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.7.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.8.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.9.zip

2021-09-09T07:38:23 [INFO] Analytics has 254,372,681 docs (target is 320,437,575)

2021-09-09T07:43:47 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2

2021-09-09T07:43:57 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2

2021-09-09T07:44:07 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2

2021-09-09T07:44:17 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2

....

2021-09-09T07:46:58 [ERROR] Request http://172.23.96.8:8095/analytics/status/ingestion/v2  failed after 20 attempts

Build: 7.1.0-1250

Job: http://perf.jenkins.couchbase.com/job/oceanus/6883/ 

Logs:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.205.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.57.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.5.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.7.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.8.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.9.zip

2021-09-09T06:11:14 [INFO] Analytics has 232,576,809 docs (target is 320,437,575)

2021-09-09T06:16:14 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2 

Traceback (most recent call last):

File "env/bin/perfrunner", line 33, in <module> sys.exit(load_entry_point('perfrunner==0.0.0', 'console_scripts', 'perfrunner')())

File "/workspace/workspace/oceanus/perfrunner/_main_.py", line 41, in main test.run()

File "/workspace/workspace/oceanus/perfrunner/tests/analytics.py", line 339, in run self.sync()

File "/workspace/workspace/oceanus/perfrunner/tests/analytics.py", line 120, in sync self.analytics_node)

File "/workspace/workspace/oceanus/perfrunner/helpers/monitor.py", line 818, in monitor_data_synced ingestion_progress = self.get_ingestion_progress(analytics_node)

File "/workspace/workspace/oceanus/perfrunner/helpers/monitor.py", line 767, in get_ingestion_progress for scope_state in stats["links"][0]["state"]:

KeyError: 'links'