Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Untriaged
-
Yes
-
KV Sprint 2020-February
Description
Pillowfight throughput test with replicate_to=1, persist_to=0. This test is being run with libcouchbase 3.0.0-alpha.5 and python sdk 3.0.0-alpha.5. Up to build 1213 we are seeing consistent throughput but in build 1214 throughput drops to 0-6k. I am not sure of the exact throughput drop since the slow down is so much, the test times out. I have logged into the machine and watched the stats for a bit and witnessed both 0 ops and ~6k ops. Throughput for this test is normally around 200k ops.
1226 - http://perf.jenkins.couchbase.com/job/ares/12287/ - Hang
1214 - http://perf.jenkins.couchbase.com/job/ares/12289/ - Hang
1213 - http://perf.jenkins.couchbase.com/job/ares/12291/ - 215601
1210 - http://perf.jenkins.couchbase.com/job/ares/12286/ - 215308
1180 - http://perf.jenkins.couchbase.com/job/ares/12285/ - 213250
Graphs are unavailable for the bad tests. Changelog doesnt indicate any obvious culprit: http://172.23.123.43:8000/getchangelog?product=couchbase-server&fromb=7.0.0-1213&tob=7.0.0-1215
On top of that, I have runs some experiments with the latest versions of libcouchbase (3.0.0) and python sdk (3.0.0-beta.3). These tests show similar slowdown but on builds that the original test performs fine:
1214 - http://perf.jenkins.couchbase.com/job/ares/12292/ - Hang
1213 - http://perf.jenkins.couchbase.com/job/ares/12293/ - Hang
1210 - http://perf.jenkins.couchbase.com/job/ares/12294/ - Hang
1180 - http://perf.jenkins.couchbase.com/job/ares/12295/ - Hang
Pillowfight command:
cbc-pillowfight --password password --batch-size 1 --num-items 20000000 --num-threads 128 --min-size 512 --max-size 512 --persist-to 0 --replicate-to 1 --json --spec "couchbase://172.23.133.13/bucket-1?durability_interval=0.02&enable_tracing=false&ipv6=allow" --set-pct 100 --num-cycles 2000000 --no-population