Details
-
Bug
-
Resolution: Cannot Reproduce
-
Test Blocker
-
7.1.0
-
Untriaged
-
1
-
Unknown
-
CX Sprint 264
Description
All analytics tests with 999 collections failed during the ingestion process. We don't see this issue in tests with 3 and 99 collections. The tests failed because (1) they didn't get response from analytics/status/ingestion/v2 after 20 attempts, or (2) the response from analytics/status/ingestion/v2 didn't have the right information.
Build: 7.1.0-1250
Job: http://perf.jenkins.couchbase.com/job/oceanus/6884/
Logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.57.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.5.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.7.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.8.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6884/172.23.96.9.zip
2021-09-09T07:38:23 [INFO] Analytics has 254,372,681 docs (target is 320,437,575)
2021-09-09T07:43:47 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2
2021-09-09T07:43:57 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2
2021-09-09T07:44:07 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2
2021-09-09T07:44:17 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2
....
2021-09-09T07:46:58 [ERROR] Request http://172.23.96.8:8095/analytics/status/ingestion/v2 failed after 20 attempts
Build: 7.1.0-1250
Job: http://perf.jenkins.couchbase.com/job/oceanus/6883/
Logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.205.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.57.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.5.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.7.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.8.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-oceanus-6883/172.23.96.9.zip
2021-09-09T06:11:14 [INFO] Analytics has 232,576,809 docs (target is 320,437,575)
2021-09-09T06:16:14 [WARNING] Retrying http://172.23.96.8:8095/analytics/status/ingestion/v2
Traceback (most recent call last):
File "env/bin/perfrunner", line 33, in <module> sys.exit(load_entry_point('perfrunner==0.0.0', 'console_scripts', 'perfrunner')())
File "/workspace/workspace/oceanus/perfrunner/_main_.py", line 41, in main test.run()
File "/workspace/workspace/oceanus/perfrunner/tests/analytics.py", line 339, in run self.sync()
File "/workspace/workspace/oceanus/perfrunner/tests/analytics.py", line 120, in sync self.analytics_node)
File "/workspace/workspace/oceanus/perfrunner/helpers/monitor.py", line 818, in monitor_data_synced ingestion_progress = self.get_ingestion_progress(analytics_node)
File "/workspace/workspace/oceanus/perfrunner/helpers/monitor.py", line 767, in get_ingestion_progress for scope_state in stats["links"][0]["state"]:
KeyError: 'links'