Details
-
Bug
-
Resolution: Duplicate
-
Major
-
3.0
-
None
-
Security Level: Public
-
couchbase-server-enterprise_x86_64_3.0.0-350-rel.deb
ubuntu 12.04, amazon m1.xlarge, 15G RAM, 4 ECU
-
Untriaged
-
Ubuntu 64-bit
Description
Memcached ops/sec can drop to 500 ops/sec from 10K ops/sec if a UPR consumer is subscribing to the cluster. This is lot of free RAM, CPU etc while this is happening.
Steps to reproduce:
1. Install couchbase-server-enterprise_x86_64_3.0.0-350-rel.deb on single node (amazon m1.xlarge, 15G RAM, 4 ecu)
2. Create default bucket
3. Start a UPR consumer which subscribes for all 1024 vbuckets for mutations.
4. Start front-end load using 2 mcsoda clients from 2 separate EC2 nodes (500K items, 30% sets, 70% gets)
client1:
./lib/perf_engines/mcsoda.py membase://10.147.4.31:8091 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=0.3 min-value-size=256,512 max-items=500000 exit-after-creates=1 prefix=70one
client2:
./lib/perf_engines/mcsoda.py membase://10.147.4.31:8091 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=0.3 min-value-size=256,512 max-items=500000 exit-after-creates=1 prefix=70two
This generates roughly 18k ops/sec as seen in screenshot normal_cluster.png
5. Repeat Step4 after changing the prefix 2-3 times. One of the clients starts getting very slow response as can be seen in througput_drop.png. The ops/sec drop to 500 from 9-10k.
6. Kill the UPR consumer while client is experiencing low ops/sec. Instantly the ops/sec surge back to 10k ops/sec as can be seen in after_killing_upr_consumer.png.
The UPR consumer is only taking 30% cpu and no memory. Its not doing any network activity as well(its running on the KV node).