Details
-
Bug
-
Resolution: Duplicate
-
Blocker
-
2.0
-
Security Level: Public
-
build 1751 on Centos 64
Description
Test to reproduce the issue:
./testrunner -i ../../ini/vm-4nodes-2.ini -t rebalance.rebalancein.RebalanceInTests.rebalance_in_with_ops,nodes_in=1,replicas=1,items=50000,doc_ops=create;update;delete,disabled_consistent_view=False
The test is rebalancing in 1 node and inserting/mutating docs while rebalance is in progress.
After rebalance is finished, ep_queue_size is always > 0 and doesn't get drained even after waiting for a very long time e.g.
2012-09-21 05:57:36,465 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:57:41,484 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:57:46,500 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:57:51,518 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:57:56,536 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:58:01,556 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:58:06,574 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:58:11,591 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
2012-09-21 05:58:16,610 - root - INFO - Not Ready: ep_queue_size 2042 == 0 expected on '10.3.2.57'
In the logs following message can be seen repeating:
memcached<0.404.0>: Fri Sep 21 06:01:40.846677 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.849414 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.852653 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.855700 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.858190 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.860517 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.863663 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.867291 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.870259 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.872901 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.876232 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
memcached<0.404.0>: Fri Sep 21 06:01:40.878934 PDT 3: Retry notify CouchDB of update, vbid=312 rev=1
Diagnostics are attached.