Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Major
Fix Version/s: 5.0.0
Affects Version/s: 4.5.1, 5.0.0
Component/s: couchbase-bucket
Labels:
- performance
Environment:
2 x E5-2680 v3 (48 vCPU), 256 GB RAM, CentOS 7

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
Build 5.0.0-1740:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.100.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.101.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.102.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.103.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.104.zip

Build 5.0.0-1665:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.100.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.101.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.102.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.103.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.104.zip

Show
Build 5.0.0-1740: https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.100.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.101.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.102.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.103.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-218/172.23.96.104.zip Build 5.0.0-1665: https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.100.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.101.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.102.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.103.zip https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-182/172.23.96.104.zip
Is this a Regression?:
Yes

Description

Setup

Source cluster: 5 nodes, destination cluster: 4 nodes.
1 node is being added to the destination cluster.
Uni-directional XDCR.
10K mixed KV ops/sec on both sides.

Problem
Rebalance makes no progress for 2 hours.

Please note, our performance tests are configured to terminate the execution after 2 hours of observed idleness. It's possible that rebalance would have finished after a longer period of time.

So far this issue happened twice. The logs are provided for both occurrences.

Attachments

Issue Links

is duplicated by

MB-21050 Intermittent failure in ep_engine test 'dcp cursor dropping backfill'

Closed

MB-23163 Rebalance deadlock with busy writer threads and no traffic

Closed

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...