[MB-36400] [high-bucket] FTS rebalance out in high bucket density test takes really long hours (28hrs+) - Couchbase database

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 6.5.0
Fix Version/s: CheshireCat.Next
Component/s: fts
Labels:
Environment:
Build 6.5.0-4380

Triage:
Untriaged
Is this a Regression?:
No

Description

Build 6.5.0-4380

Observed that rebalancing out FTS node in high bucket density test with 30 buckets taking 764.42 min and fails with following reason.

Each bucket have 1 FTS index.

Rebalance failure reason-

Rebalance exited with reason {service_rebalance_failed,fts,

{worker_died,

{'EXIT',<0.13606.525>,

{{linked_process_died,<27016.24380.381>,

{timeout,

{gen_server,call,

[<27016.22501.381>,

{call,"ServiceAPI.GetCurrentTopology",

#Fun<json_rpc_connection.0.102434519>},

60000]}}},

{gen_server,call,

[{'service_agent-fts','ns_1@172.23.97.15'},

{if_rebalance,<0.19462.525>,

{start_rebalance,

<<"f81721335b42662e837612d6b366cca3">>,

rebalance,

[{[{node_id,

<<"827b0f5298e3447fa469c0ebdcad3da0">>},

{priority,0},

{opaque,null}],

full}],

[[{node_id,

<<"592321897f0f721188d778a77f4c9d9b">>},

{priority,0},

{opaque,null}]],

<0.13606.525>}},

90000]}}}}}.

Rebalance Operation Id = f5116259df628e7619516c5b6cba1997

On build 6.5.0-2082, this rebalance out took 79 min.

Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/322/

Logs-
FTS node- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.20.zip
FTS node going out- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.15.zip

Attachments

Issue Links

relates to

MB-34911 [high-bucket] FTS: Rebalance failed after failover in high bucket density test

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

Ascending order - Click to sort in descending order

Mahesh Mandhare (Inactive) created issue - 09/Oct/19 3:13 AM

Mahesh Mandhare (Inactive) made changes - 09/Oct/19 3:17 AM

Field	Original Value	New Value
Summary	FTS rebalance out in high bucket density test takes huge time	FTS rebalance out in high bucket density test takes 12 hours

Mahesh Mandhare (Inactive) made changes - 09/Oct/19 3:18 AM

Description

Mahesh Mandhare (Inactive) made changes - 09/Oct/19 3:52 AM

Summary

FTS rebalance out in high bucket density test takes 12 hours

FTS rebalance out in high bucket density test takes 12 hours and fails

Mahesh Mandhare (Inactive) made changes - 09/Oct/19 5:44 AM

Description

Build 6.5.0-4380

Observed that rebalancing out FTS node in high bucket density test with 30 buckets taking 764.42 min and fails with following reason.

Each bucket have 1 FTS index.

Rebalance failure reason-
{noformat}Rebalance exited with reason {service_rebalance_failed,fts,
{worker_died,
{'EXIT',<0.13606.525>,
{{linked_process_died,<27016.24380.381>,
{timeout,
{gen_server,call,
[<27016.22501.381>,
{call,"ServiceAPI.GetCurrentTopology",
#Fun<json_rpc_connection.0.102434519>},
60000]}}},
{gen_server,call,
[{'service_agent-fts','ns_1@172.23.97.15'},
{if_rebalance,<0.19462.525>,
{start_rebalance,
<<"f81721335b42662e837612d6b366cca3">>,
rebalance,
[{[{node_id,
<<"827b0f5298e3447fa469c0ebdcad3da0">>},
{priority,0},
{opaque,null}],
full}],
[[{node_id,
<<"592321897f0f721188d778a77f4c9d9b">>},
{priority,0},
{opaque,null}]],
<0.13606.525>}},
90000]}}}}}.
Rebalance Operation Id = f5116259df628e7619516c5b6cba1997
{noformat}
On build 6.5.0-2082, this rebalance out took 79 min.

Job- [http://perf.jenkins.couchbase.com/job/arke-multi-bucket/322/]

Logs-
FTS node- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.20.zip
FTS node going out- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.15.zip

Other nodes-
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.16.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.17.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.96.23.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.12.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.13.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.14.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.177.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.19.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-322/172.23.97.20.zip

Wayne Siu made changes - 09/Oct/19 4:56 PM

Summary

FTS rebalance out in high bucket density test takes 12 hours and fails

[high-bucket] FTS rebalance out in high bucket density test takes 12 hours and fails

Raju Suravarjjala made changes - 09/Oct/19 7:23 PM

Priority

Major [ 3 ]

Critical [ 2 ]

Raju Suravarjjala made changes - 09/Oct/19 7:23 PM

Fix Version/s

Mad-Hatter [ 15037 ]

Sreekanth Sivasankaran made changes - 14/Oct/19 8:43 PM

Assignee

Keshav Murthy [ keshav ]

Sreekanth Sivasankaran [ sreekanth sivasankaran ]

Sreekanth Sivasankaran made changes - 14/Oct/19 8:46 PM

Assignee

Sreekanth Sivasankaran [ sreekanth sivasankaran ]

Mahesh Mandhare [ mahesh.mandhare ]

Lynn Straus made changes - 24/Oct/19 10:24 AM

Labels

Performance high-bucket-density

Performance approved-for-mad-hatter high-bucket-density

Mahesh Mandhare (Inactive) made changes - 25/Oct/19 2:47 AM

Assignee

Mahesh Mandhare [ mahesh.mandhare ]

Sreekanth Sivasankaran [ sreekanth sivasankaran ]

Lynn Straus made changes - 29/Oct/19 3:09 PM

Due Date

15/Nov/19

Sreekanth Sivasankaran made changes - 30/Oct/19 3:48 AM

Is this a Regression?

Yes [ 10450 ]

No [ 10451 ]

Keshav Murthy made changes - 30/Oct/19 9:11 AM

Fix Version/s		Cheshire-Cat [ 15915 ]
Fix Version/s	Mad-Hatter [ 15037 ]

Sreekanth Sivasankaran made changes - 13/Jan/20 9:26 AM

Summary

[high-bucket] FTS rebalance out in high bucket density test takes 12 hours and fails

[high-bucket] FTS rebalance out in high bucket density test takes really long hours (1

Sreekanth Sivasankaran made changes - 13/Jan/20 9:26 AM

Summary

[high-bucket] FTS rebalance out in high bucket density test takes really long hours (1

[high-bucket] FTS rebalance out in high bucket density test takes really long hours (28hrs+)

Abhinav Dangeti made changes - 22/Jan/20 2:51 PM

Link

This issue relates to ~~MB-34911~~ [ ~~MB-34911~~ ]

Keshav Murthy made changes - 26/Jan/21 3:23 PM

Assignee

Sreekanth Sivasankaran [ sreekanth sivasankaran ]

Jyotsna Nayak [ jyotsna.nayak ]

Keshav Murthy made changes - 19/Feb/21 10:12 AM

Priority

Critical [ 2 ]

Major [ 3 ]

Lynn Straus made changes - 10/Mar/21 3:06 PM

Fix Version/s		CheshireCat.Next [ 16908 ]
Fix Version/s	Cheshire-Cat [ 15915 ]
Labels	Performance approved-for-mad-hatter high-bucket-density	Performance approved-for-mad-hatter deferred-from-Cheshire-Cat high-bucket-density

Jyotsna Nayak made changes - 12/Mar/21 9:46 AM

Status

Open [ 1 ]

In Progress [ 3 ]

Jyotsna Nayak made changes - 14/Dec/21 8:06 AM

Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

People

Assignee:: Jyotsna Nayak

Reporter:: Mahesh Mandhare (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Due:: 15/Nov/19

Created:: 09/Oct/19 3:13 AM

Updated:: 14/Dec/21 8:06 AM

Resolved:: 14/Dec/21 8:06 AM

Gerrit Reviews

There are no open Gerrit changes

Couchbase Server

[high-bucket] FTS rebalance out in high bucket density test takes really long hours (28hrs+)

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty