Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 3.0-Beta, 3.0
Affects Version/s: 2.5.1, 3.0
Component/s: XDCR
Security Level: Public
Labels:
None
Environment:
Build 3.0.0-1096 build

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
[Source]
10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-11873/801b4078/10.5.2.228-822014-328-couch.tar.gz
10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-11873/b217005f/10.5.2.228-822014-331-diag.zip
10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-11873/ea66d27e/10.5.2.228-diag.txt.gz
10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-11873/92160eb7/10.5.2.229-822014-328-couch.tar.gz
10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-11873/9a31d920/10.5.2.229-822014-332-diag.zip
10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-11873/f6bd6c25/10.5.2.229-diag.txt.gz
10.5.2.230 : https://s3.amazonaws.com/bugdb/jira/MB-11873/7c102842/10.5.2.230-822014-333-diag.zip
10.5.2.230 : https://s3.amazonaws.com/bugdb/jira/MB-11873/c30820dd/10.5.2.230-822014-328-couch.tar.gz
10.5.2.230 : https://s3.amazonaws.com/bugdb/jira/MB-11873/d1967b13/10.5.2.230-diag.txt.gz

[Destination]
10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-11873/06eb3400/10.5.2.231-822014-334-diag.zip
10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-11873/b3696a88/10.5.2.231-diag.txt.gz
10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-11873/e2766714/10.5.2.231-822014-329-couch.tar.gz
10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-11873/4cb74f5f/10.5.2.232-diag.txt.gz
10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-11873/9647ecd1/10.5.2.232-822014-329-couch.tar.gz
10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-11873/ab5d9139/10.5.2.232-822014-335-diag.zip
10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-11873/1be82c8f/10.5.2.233-822014-329-couch.tar.gz
10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-11873/257121e9/10.5.2.233-diag.txt.gz
10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-11873/7cde338c/10.5.2.233-822014-336-diag.zip
10.5.2.234 : https://s3.amazonaws.com/bugdb/jira/MB-11873/311d2a96/10.5.2.234-diag.txt.gz
10.5.2.234 : https://s3.amazonaws.com/bugdb/jira/MB-11873/34b30ec5/10.5.2.234-822014-329-couch.tar.gz
10.5.2.234 : https://s3.amazonaws.com/bugdb/jira/MB-11873/40000b0f/10.5.2.234-822014-337-diag.zip
10.3.5.68 : https://s3.amazonaws.com/bugdb/jira/MB-11873/97470200/10.3.5.68-822014-329-couch.tar.gz
10.3.5.68 : https://s3.amazonaws.com/bugdb/jira/MB-11873/aae79e46/10.3.5.68-822014-337-diag.zip
10.3.5.68 : https://s3.amazonaws.com/bugdb/jira/MB-11873/cdc8e5d1/10.3.5.68-diag.txt.gz

Show
[Source] 10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-11873/801b4078/10.5.2.228-822014-328-couch.tar.gz 10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-11873/b217005f/10.5.2.228-822014-331-diag.zip 10.5.2.228 : https://s3.amazonaws.com/bugdb/jira/MB-11873/ea66d27e/10.5.2.228-diag.txt.gz 10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-11873/92160eb7/10.5.2.229-822014-328-couch.tar.gz 10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-11873/9a31d920/10.5.2.229-822014-332-diag.zip 10.5.2.229 : https://s3.amazonaws.com/bugdb/jira/MB-11873/f6bd6c25/10.5.2.229-diag.txt.gz 10.5.2.230 : https://s3.amazonaws.com/bugdb/jira/MB-11873/7c102842/10.5.2.230-822014-333-diag.zip 10.5.2.230 : https://s3.amazonaws.com/bugdb/jira/MB-11873/c30820dd/10.5.2.230-822014-328-couch.tar.gz 10.5.2.230 : https://s3.amazonaws.com/bugdb/jira/MB-11873/d1967b13/10.5.2.230-diag.txt.gz [Destination] 10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-11873/06eb3400/10.5.2.231-822014-334-diag.zip 10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-11873/b3696a88/10.5.2.231-diag.txt.gz 10.5.2.231 : https://s3.amazonaws.com/bugdb/jira/MB-11873/e2766714/10.5.2.231-822014-329-couch.tar.gz 10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-11873/4cb74f5f/10.5.2.232-diag.txt.gz 10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-11873/9647ecd1/10.5.2.232-822014-329-couch.tar.gz 10.5.2.232 : https://s3.amazonaws.com/bugdb/jira/MB-11873/ab5d9139/10.5.2.232-822014-335-diag.zip 10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-11873/1be82c8f/10.5.2.233-822014-329-couch.tar.gz 10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-11873/257121e9/10.5.2.233-diag.txt.gz 10.5.2.233 : https://s3.amazonaws.com/bugdb/jira/MB-11873/7cde338c/10.5.2.233-822014-336-diag.zip 10.5.2.234 : https://s3.amazonaws.com/bugdb/jira/MB-11873/311d2a96/10.5.2.234-diag.txt.gz 10.5.2.234 : https://s3.amazonaws.com/bugdb/jira/MB-11873/34b30ec5/10.5.2.234-822014-329-couch.tar.gz 10.5.2.234 : https://s3.amazonaws.com/bugdb/jira/MB-11873/40000b0f/10.5.2.234-822014-337-diag.zip 10.3.5.68 : https://s3.amazonaws.com/bugdb/jira/MB-11873/97470200/10.3.5.68-822014-329-couch.tar.gz 10.3.5.68 : https://s3.amazonaws.com/bugdb/jira/MB-11873/aae79e46/10.3.5.68-822014-337-diag.zip 10.3.5.68 : https://s3.amazonaws.com/bugdb/jira/MB-11873/cdc8e5d1/10.3.5.68-diag.txt.gz
Is this a Regression?:
Yes

Description

http://qa.hq.northscale.net/view/3.0%20By%20COMPONENT/job/centos_x64--107_01--rebalanceXDCR-P1/39/consoleFull

[Test]
./testrunner ~~i centos_x64107_01~~-rebalanceXDCR-P1.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=unidirection,ctopology=chain,doc-ops=update-delete,rebalance=destination,num_rebalance=2,GROUP=P0;xmem

[Test Error logs]
2014-08-02 03:22:12 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 91121 == 70000 expected on '10.5.2.231:8091''10.5.2.232:8091''10.5.2.233:8091''10.5.2.234:8091''10.3.5.68:8091', default bucket
2014-08-02 03:22:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_active_curr_items 91121 == 70000 expected on '10.5.2.231:8091''10.5.2.232:8091''10.5.2.233:8091''10.5.2.234:8091''10.3.5.68:8091', default bucket
2014-08-02 03:22:13 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 91121 == 70000 expected on '10.5.2.231:8091''10.5.2.232:8091''10.5.2.233:8091''10.5.2.234:8091''10.3.5.68:8091', default bucket

[Test Steps]
1. Setup Uni-XDCR Source (3 Nodes) Destination ( 3 Nodes), Bucket: default, Mode: CAPI.
2. Load 1M items on Source node.
3. Add 2 nodes at Destination Cluster. Wait for rebalance to finish.
4. Update 30K and Delete 30 K items on Source.
5. Source has 70000K as expected.
6. Destination Count: 91121 expected: 70000. (Mutations are not replicated well).