Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
2.0.1, 2.1.0, 2.1.1, 2.2.0
-
Security Level: Public
-
PCI Team - Sprint 3, PCI Team - Sprint 4, PCI Team - Sprint 5, PCI Team - Sprint 6, CB Bucket - 08/12 - 08/29, 12/Aug - 30/Aug
Description
Environment:
-memcache-test running through client-side Moxi.
-memcache-test workload: /root/mctest/Users/ingenthr/tmp/memcachetest-centos_x86 -h localhost -i 1000 -t 4 -K test -M 10 -F -S -l
-Moxi config: CLI: /opt/moxi/bin/moxi -u nobody -Z usr=Administrator,pwd=password http://10.197.1.147:8091/pools/default/bucketsStreaming/beer-sample -vv -O /var/log/moxi,
config:
port_listen=11211,
default_bucket_name=default,
downstream_max=1024,
downstream_conn_max=4,
downstream_conn_queue_timeout=200,
downstream_timeout=5000,
wait_queue_timeout=200,
connect_max_errors=5,
connect_retry_interval=30000,
connect_timeout=400,
auth_timeout=100,
cycle=200
-4 nodes of Couchbase Server 2.0.1. c1.medium EC2 instances, beer-sample database bucket
-2 nodes of same config added to cluster and rebalanced
Observation:
-Workload very steady around 8k ops/sec before rebalance
-Workload drops to around 4k ops/sec during rebalance. Both gets and sets affected, compaction not running when load initially drops (kicks in eventually). Workload very choppy during rebalance.
-Workload returns to 8k ops/sec after rebalance
Client memcachetest and moxi logs:
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client1.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/client2.zip
Cluster logs:
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node1.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node2.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node3.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node4.zip
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node5.zip (aded to first 4)
https://s3.amazonaws.com/customers.couchbase.com/demo_performance/node6.zip (added to first 4)