Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
6.5.0
-
Build 6.5.0-4380
-
Untriaged
-
No
Description
Build 6.5.0-4380
Observed that rebalancing out FTS node in high bucket density test with 30 buckets taking 764.42 min and fails with following reason.
Each bucket have 1 FTS index.
Rebalance failure reason-
Rebalance exited with reason {service_rebalance_failed,fts,
|
{worker_died,
|
{'EXIT',<0.13606.525>,
|
{{linked_process_died,<27016.24380.381>,
|
{timeout,
|
{gen_server,call,
|
[<27016.22501.381>,
|
{call,"ServiceAPI.GetCurrentTopology",
|
#Fun<json_rpc_connection.0.102434519>},
|
60000]}}},
|
{gen_server,call,
|
[{'service_agent-fts','ns_1@172.23.97.15'},
|
{if_rebalance,<0.19462.525>,
|
{start_rebalance,
|
<<"f81721335b42662e837612d6b366cca3">>,
|
rebalance,
|
[{[{node_id,
|
<<"827b0f5298e3447fa469c0ebdcad3da0">>},
|
{priority,0},
|
{opaque,null}],
|
full}],
|
[[{node_id,
|
<<"592321897f0f721188d778a77f4c9d9b">>},
|
{priority,0},
|
{opaque,null}]],
|
<0.13606.525>}},
|
90000]}}}}}.
|
Rebalance Operation Id = f5116259df628e7619516c5b6cba1997
|
 On build 6.5.0-2082, this rebalance out took 79 min.
Job-Â http://perf.jenkins.couchbase.com/job/arke-multi-bucket/322/
Logs-
FTS node-Â https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.20.zip
FTS node going out-Â https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.15.zip
Other nodes-
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.16.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.17.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.23.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.12.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.13.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.14.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.177.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.19.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.20.zip
Attachments
Issue Links
- relates to
-
MB-34911 [high-bucket] FTS: Rebalance failed after failover in high bucket density test
-
- Closed
-
Activity
Field | Original Value | New Value |
---|---|---|
Summary | FTS rebalance out in high bucket density test takes huge time | FTS rebalance out in high bucket density test takes 12 hours |
Summary | FTS rebalance out in high bucket density test takes 12 hours | FTS rebalance out in high bucket density test takes 12 hours and fails |
Summary | FTS rebalance out in high bucket density test takes 12 hours and fails | [high-bucket] FTS rebalance out in high bucket density test takes 12 hours and fails |
Priority | Major [ 3 ] | Critical [ 2 ] |
Fix Version/s | Mad-Hatter [ 15037 ] |
Assignee | Keshav Murthy [ keshav ] | Sreekanth Sivasankaran [ sreekanth sivasankaran ] |
Assignee | Sreekanth Sivasankaran [ sreekanth sivasankaran ] | Mahesh Mandhare [ mahesh.mandhare ] |
Labels | Performance high-bucket-density | Performance approved-for-mad-hatter high-bucket-density |
Assignee | Mahesh Mandhare [ mahesh.mandhare ] | Sreekanth Sivasankaran [ sreekanth sivasankaran ] |
Due Date | 15/Nov/19 |
Is this a Regression? | Yes [ 10450 ] | No [ 10451 ] |
Fix Version/s | Cheshire-Cat [ 15915 ] | |
Fix Version/s | Mad-Hatter [ 15037 ] |
Summary | [high-bucket] FTS rebalance out in high bucket density test takes 12 hours and fails | [high-bucket] FTS rebalance out in high bucket density test takes really long hours (1 |
Summary | [high-bucket] FTS rebalance out in high bucket density test takes really long hours (1 | [high-bucket] FTS rebalance out in high bucket density test takes really long hours (28hrs+) |
Assignee | Sreekanth Sivasankaran [ sreekanth sivasankaran ] | Jyotsna Nayak [ jyotsna.nayak ] |
Priority | Critical [ 2 ] | Major [ 3 ] |
Fix Version/s | CheshireCat.Next [ 16908 ] | |
Fix Version/s | Cheshire-Cat [ 15915 ] | |
Labels | Performance approved-for-mad-hatter high-bucket-density | Performance approved-for-mad-hatter deferred-from-Cheshire-Cat high-bucket-density |
Status | Open [ 1 ] | In Progress [ 3 ] |
Resolution | Fixed [ 1 ] | |
Status | In Progress [ 3 ] | Closed [ 6 ] |