Details
-
Bug
-
Resolution: Incomplete
-
Critical
-
3.0-Beta
-
Security Level: Public
-
None
-
3.0 build 1105
-
Untriaged
-
Unknown
-
June 30 - July 18
Description
Test:
- 4 nodes, 4 buckets (replica=3)
- load each bucket to 30% dgm
- rebalance out 2 nodes with ops running on buckets
(target op-rate per bucket is 10k. load ratio = set:10% update:60% , get:25% (5% miss rate) delete:5%
each node has been at 0% for about 40 minutes now with 0 keys transferred
from debug.log:
rebalance:debug,2014-08-08T14:13:06.838,ns_1@172.23.106.53:<0.17024.0>:janitor_agent:do_wait_seqno_persisted:1098]Got etmpfail waiting for seq no persistence. Will try again
[rebalance:debug,2014-08-08T14:13:06.874,ns_1@172.23.106.53:<0.19676.0>:janitor_agent:do_wait_seqno_persisted:1098]Got etmpfail waiting for seq no persistence. Will try again
[rebalance:debug,2014-08-08T14:13:06.994,ns_1@172.23.106.53:<0.19596.0>:janitor_agent:do_wait_seqno_persisted:1098]Got etmpfail waiting for seq no persistence. Will try again
[rebalance:debug,2014-08-08T14:13:07.101,ns_1@172.23.106.53:<0.19516.0>:janitor_agent:do_wait_seqno_persisted:1098]Got etmpfail waiting for seq no persistence. Will try again
memcached log snippet:
Fri Aug 8 14:13:21.678937 PDT 3: (standardbucket3) Notified the timeout on checkpoint persistence for vbucket 313, cookie 0x591e600
Fri Aug 8 14:13:21.742580 PDT 3: (standardbucket3) Notified the timeout on checkpoint persistence for vbucket 317, cookie 0x5729600
Fri Aug 8 14:13:27.263487 PDT 3: (standardbucket3) Notified the timeout on checkpoint persistence for vbucket 321, cookie 0x596dc00
Fri Aug 8 14:13:27.301718 PDT 3: (standardbucket3) Notified the timeout on checkpoint persistence for vbucket 325, cookie 0x584d300
the scenario passed with 2 replica's and rebalancing out 1 node.