Details
-
Task
-
Resolution: Unresolved
-
Major
-
None
-
None
-
None
Description
What is the issue?
Currently the Examinador tests are quite unstable (probably for more than one reason).
What needs to be done?
We need to improve stability of our tests because re-verifying commits on gerrit when you know that there are no issues with the code is quite time-consuming and frustrating.
Observed issues:
- TimeoutExpired when running "cbbackupmgr backup" [STATUS: Unknown, Investigating]
http://cv.jenkins.couchbase.com/job/Examinador%20cv%20multi-branch/job/master/545/robot/
- Test suite setup fails with Parent suite setup failed: TimeoutException: <RC=0xC9[LCB_ERR_TIMEOUT (201)] [STATUS: Fixed, Transient issue]
http://cv.jenkins.couchbase.com/job/Examinador%20cv%20multi-branch/job/master/513/robot/
- Test that checks eventing restore fails [STATUS: Merged a second potential fix, Observing, Investigated]
http://cv.jenkins.couchbase.com/job/backup_integration_tests/job/master/927/
http://cv.jenkins.couchbase.com/job/backup_integration_tests/job/master/928/
Tried reproducing locally, couldn't get a run of that specific test suite to fail like in the example build. This seems to be a new issue since I couldn't find any older runs of that test suite where this specific test failed like that.
[24.01.2022] Seems to be broken again, will try increasing the delay again.
- Indexes failing to be built in 60 seconds [STATUS: Merged a potential fix, Success, Investigated]
http://cv.jenkins.couchbase.com/job/Examinador%20cv%20multi-branch/job/master/495/robot/
This might be related to the fact that I decreased the time after which indexes will be considered to fail to build from 120 seconds to 60 seconds in https://github.com/couchbaselabs/examinador/commit/d8d71a04796332cdf424a6bf661056cffb2bb0a8. Might be worth increasing that back to 120 seconds as it seems that it does occasionally take a while to build them from my testing.
- Tests that check cbimport, cbexport and cbdocloader [STATUS: Not an issue]
After adding new cbimport, cbexport and cbdocloader tests, which create and delete buckets quite frequently and work with sizeable sample datasets, our tests have become unstable in a sense that some of the new tests might occasionally fail because the cluster is not in a ready state after one of them (or some other timings issues, which needs investigating).
Attachments
Gerrit Reviews
For Gerrit Dashboard: MB-50144 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
168820,4 | MB-50144 Increase index building timeout | master | examinador | Status: MERGED | +2 | +1 |
168821,4 | MB-50144 Add delay after restoring eventing function | master | examinador | Status: MERGED | +2 | +1 |
169338,3 | MB-50144 Increase delay after restoring eventing function | master | examinador | Status: MERGED | +2 | +1 |
171641,7 | MB-50144 Fix eventing tests failing, improve all functions status check | master | examinador | Status: MERGED | +2 | +1 |