Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Test Blocker
Fix Version/s: 4.0.0
Affects Version/s: 4.0.0
Component/s: XDCR
Security Level: Public
Labels:
None
Environment:
CentOS 6.x, build 995, 2 node clusters, checkpoint_interval=1800

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
http://qa.sc.couchbase.com/view/BY%20COMPONENT/job/cen006-p0-xxdcr-vsetS2-01-goxdcr-rebalance/ws/logs/testrunner-15-Jan-22_13-25-11/test_1/

Show
http://qa.sc.couchbase.com/view/BY%20COMPONENT/job/cen006-p0-xxdcr-vsetS2-01-goxdcr-rebalance/ws/logs/testrunner-15-Jan-22_13-25-11/test_1/
Is this a Regression?:
No

Description

Testcase:
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,replication_type=xmem,get-cbcollect-info=True -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=unidirection,ctopology=chain,update=C1,delete=C1,rebalance=C2,num_rebalance=2,GROUP=P0;xmem

http://qa.sc.couchbase.com/view/BY%20COMPONENT/job/cen006-p0-xxdcr-vsetS2-01-goxdcr-rebalance/3/consoleFull see test#1

[C1]
'172.23.105.57:8091''172.23.105.62:8091'

[C2]
172.23.105.60:8091''172.23.105.63:8091'' <--- 172.23.105.61:8091 (rebalanced-in node)

2015-01-22 13:29:21 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: vb_replica_curr_items 100000 == 70000 expected on '172.23.105.60:8091''172.23.105.63:8091''172.23.105.61:8091', default bucket
2015-01-22 13:29:27 | WARNING | MainProcess | Cluster_Thread | [task.check] Not Ready: curr_items 100000 == 70000 expected on '172.23.105.60:8091''172.23.105.63:8091''172.23.105.61:8091', default bucket

{'deleted': 0, 'seqno': 2, 'cas': 1421962264395120640, 'flags': 0, 'expiration': 0}

2015-01-22 13:31:14 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 1421962141790109696, 'flags': 0, 'expiration': 0}

{'deleted': 0, 'seqno': 2, 'cas': 1421962300904112128, 'flags': 0, 'expiration': 0}

2015-01-22 13:31:14 | ERROR | MainProcess | load_gen_task | [task._check_key_revId] Dest meta data:

{'deleted': 0, 'seqno': 1, 'cas': 1421962148093755392, 'flags': 0, 'expiration': 0}

Link to logs attached.

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Aruna Piravi (Inactive)

Reporter:: Aruna Piravi (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 22/Jan/15 2:06 PM

Updated:: 28/Jan/15 5:35 PM

Resolved:: 28/Jan/15 12:05 PM

Gerrit Reviews

There are no open Gerrit changes

Show There are 6 closed Gerrit changes

Hide There are 6 closed Gerrit changes

# This is a combination of 2 commits. # The first commit's message is:: Gerrit Review:

remove xdcr transporter since it is no longer needed after cbauth fix: Gerrit Review:

bug fix - MB-13189, MB-13190, MB-13185 1. change the pipeline stopping routine from sequential to parallel, when a part fails to stop within timeout period, proceed and leave it to die, no longer blocking and retry in such case 2. fix the problem of ReplicationManager.stop could end up nested 3. refactor the code of network connection issue detection and management into a separate struct in xmem nozzle - detect long, non-fruitful effort of trying to receive memcached response, which can be a result of network connection ruin. This can cause replication stuck - make sure suspected unhealthy connection not return to the connection pool 4. fix the pip e contruction logic. Before it only contructs one router, then all DcpNozzles and XmemNozzles are connected to it, which results the mutations from different DcpNozzle need to line up and wait for its turn to be routed, in this case, if one XmemNozzle is under perform and its queue is full, other XmemNozzle would be in starving state. Now change it to have one router per DcpNozzle. 5. adding more diagnostic information: Gerrit Review:

bug fix - MB-13189, MB-13190, MB-13185 1. change the pipeline stopping routine from sequential to parallel, when a part fails to stop within timeout period, proceed and leave it to die, no longer blocking and retry in such case 2. fix the problem of ReplicationManager.stop could end up nested 3. refactor the code of network connection issue detection and management into a separate struct in xmem nozzle - detect long, non-fruitful effort of trying to receive memcached response, which can be a result of network connection ruin. This can cause replication stuck - make sure suspected unhealthy connection not return to the connection pool 4. fix the pip e contruction logic. Before it only contructs one router, then all DcpNozzles and XmemNozzles are connected to it, which results the mutations from different DcpNozzle need to line up and wait for its turn to be routed, in this case, if one XmemNozzle is under perform and its queue is full, other XmemNozzle would be in starving state. Now change it to have one router per DcpNozzle. 5. adding more diagnostic information remove xdcr transporter since it is no longer needed after cbauth fix: Gerrit Review:

bug fix - MB-13189, MB-13190, MB-13185 1. change the pipeline stopping routine from sequential to parallel, when a part fails to stop within timeout period, proceed and leave it to die, no longer blocking and retry in such case 2. fix the problem of ReplicationManager.stop could end up nested 3. refactor the code of network connection issue detection and management into a separate struct in xmem nozzle - detect long, non-fruitful effort of trying to receive memcached response, which can be a result of network connection ruin. This can cause replication stuck - make sure suspected unhealthy connection not return to the connection pool 4. fix the pip e contruction logic. Before it only contructs one router, then all DcpNozzles and XmemNozzles are connected to it, which results the mutations from different DcpNozzle need to line up and wait for its turn to be routed, in this case, if one XmemNozzle is under perform and its queue is full, other XmemNozzle would be in starving state. Now change it to have one router per DcpNozzle. 5. adding more diagnostic information: Gerrit Review:

MB-13185 - GoXDCR: After any topology change at source/destination, updates/deletes are not replicated - checkpoint manager should not report pipeline is broken if _pre_replicate returns error. In that situation, it should start the replication with seqno = 0 - _pre_replication CAPI rest call actually requires caller be topology aware. The caller has to know which vb is on which node, then call _pre_replicate on that node. My old implementation assumes that CAPI call is topology transparent, or at cluster level like other rest calls.: Gerrit Review:

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty