Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
7.1.0
-
Untriaged
-
1
-
Unknown
-
KV 2022-Feb
Description
A number of our durability latency tests have been getting stuck during manually triggered compaction, with the Jenkins build timing out after 10 hours:
http://perf.jenkins.couchbase.com/job/ares/26739/
http://perf.jenkins.couchbase.com/job/ares/26740/
http://perf.jenkins.couchbase.com/job/ares/26741/
http://perf.jenkins.couchbase.com/job/ares/26742/
http://perf.jenkins.couchbase.com/job/ares/26743/ (cbcollects from this are attached)
The tests are running build 7.1.0-2284. The tests perform their load phase of 20M docs (with 2 replicas configured) and then get stuck during compaction which we manually trigger afterwards. I've attached cbcollect logs collected from build 26743 above, while in its stuck state.
It appears to be a product issue rather than a perf-testing issue. When re-running build 26739 from above using 7.1.0-2223, the test successfully compacts the bucket after the load phase: http://perf.jenkins.couchbase.com/job/ares/26744/console That being said, there is not a very obvious pattern regarding which tests have been affected.
In the affected tests, we see this type of compaction progress:
|
Compaction progress |
18:33:16 2022-02-13T10:33:16 [INFO] bucket_compaction: 18%, bucket: bucket-1, ddoc: None
|
18:33:18 2022-02-13T10:33:18 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:18 2022-02-13T10:33:18 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:20 2022-02-13T10:33:20 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:20 2022-02-13T10:33:20 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:22 2022-02-13T10:33:22 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:22 2022-02-13T10:33:22 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:24 2022-02-13T10:33:24 [INFO] bucket_compaction: 20%, bucket: bucket-1, ddoc: None
|
18:33:24 2022-02-13T10:33:24 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:26 2022-02-13T10:33:26 [INFO] bucket_compaction: 20%, bucket: bucket-1, ddoc: None
|
18:33:26 2022-02-13T10:33:26 [INFO] bucket_compaction: 19%, bucket: bucket-1, ddoc: None
|
18:33:28 2022-02-13T10:33:28 [INFO] bucket_compaction: 21%, bucket: bucket-1, ddoc: None
|
18:33:28 2022-02-13T10:33:28 [INFO] bucket_compaction: 20%, bucket: bucket-1, ddoc: None
|
18:33:30 2022-02-13T10:33:30 [INFO] bucket_compaction: 21%, bucket: bucket-1, ddoc: None
|
18:33:30 2022-02-13T10:33:30 [INFO] bucket_compaction: 20%, bucket: bucket-1, ddoc: None
|
18:33:32 2022-02-13T10:33:32 [INFO] bucket_compaction: 21%, bucket: bucket-1, ddoc: None
|
18:33:32 2022-02-13T10:33:32 [INFO] bucket_compaction: 20%, bucket: bucket-1, ddoc: None
|
18:33:34 2022-02-13T10:33:34 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:34 2022-02-13T10:33:34 [INFO] bucket_compaction: 22%, bucket: bucket-1, ddoc: None
|
18:33:36 2022-02-13T10:33:36 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:36 2022-02-13T10:33:36 [INFO] bucket_compaction: 22%, bucket: bucket-1, ddoc: None
|
18:33:38 2022-02-13T10:33:38 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:38 2022-02-13T10:33:38 [INFO] bucket_compaction: 26%, bucket: bucket-1, ddoc: None
|
18:33:40 2022-02-13T10:33:40 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:40 2022-02-13T10:33:40 [INFO] bucket_compaction: 26%, bucket: bucket-1, ddoc: None
|
18:33:42 2022-02-13T10:33:42 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:42 2022-02-13T10:33:42 [INFO] bucket_compaction: 26%, bucket: bucket-1, ddoc: None
|
18:33:44 2022-02-13T10:33:44 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:44 2022-02-13T10:33:44 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:46 2022-02-13T10:33:46 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:46 2022-02-13T10:33:46 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:48 2022-02-13T10:33:48 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:48 2022-02-13T10:33:48 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
18:33:50 2022-02-13T10:33:50 [INFO] bucket_compaction: 4%, bucket: bucket-1, ddoc: None
|
Compaction may start making steady progress and then compaction progress is reported to fluctuate and then it settles at a small % progress value and stays stuck like that.
Attachments
Issue Links
- duplicates
-
-
- Closed
-