Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 5.0.0
Affects Version/s: 5.0.0
Component/s: fts
Labels:
- functional-test

Triage:
Untriaged
Is this a Regression?:
Yes

Description

Build
5.0.0-3180

Is a regression from 3153.

Of 37 tests in the suite, 34 failed. This rebalance was first stuck here -

Testcase
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D+F:F:F,GROUP=P0 -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,items=1000,cluster=D,F,standard_buckets=3,sasl_buckets=3,index_per_bucket=3,update=True,expires=30,max_partitions_pindex=128,GROUP=P0

The above test creates 21 indexes with 8 pindexes each. Indexing goes fine. In tear down phase(after indexes were deleted), for 8+ mins rebalance-out(of 2 fts nodes) was stuck after which we force-removed the nodes, resulting in a series of failed rebalances.

[2017-06-22 21:34:15,008] - [rest_client:133] INFO - node 172.23.107.187 existing buckets : []

[2017-06-22 21:34:15,043] - [cluster_helper:254] INFO - rebalancing all nodes in order to remove nodes

[2017-06-22 21:34:15,047] - [rest_client:1395] INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.107.145,ns_1@172.23.107.234', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.107.187,ns_1@172.23.107.145,ns_1@172.23.107.234'}

[2017-06-22 21:34:15,055] - [rest_client:1400] INFO - rebalance operation started

[2017-06-22 21:34:15,062] - [rest_client:1548] INFO - rebalance percentage : 0.00 %

[2017-06-22 21:34:25,081] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:34:35,106] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:34:45,128] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:34:55,146] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:35:05,162] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:35:15,185] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:35:25,200] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:35:35,217] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:35:45,234] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:35:55,250] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:36:05,265] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:36:15,279] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:36:25,295] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:36:35,311] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:36:45,327] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:36:55,343] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:37:05,359] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:37:15,375] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:37:25,391] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:37:35,406] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:37:45,415] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:37:55,429] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:38:05,444] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:38:15,461] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:38:25,484] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:38:35,503] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:38:45,523] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:38:55,538] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:39:05,554] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:39:15,569] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:39:25,584] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:39:35,599] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:39:45,619] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:39:55,636] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:40:05,651] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:40:15,665] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:40:25,682] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:40:35,696] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:40:45,711] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:40:55,726] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:41:05,741] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:41:15,759] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:41:25,774] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:41:35,790] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:41:45,806] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:41:55,822] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:42:05,837] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:42:15,853] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:42:25,868] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:42:35,887] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:42:45,905] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:42:55,922] - [rest_client:1548] INFO - rebalance percentage : 17.97 %

[2017-06-22 21:42:55,922] - [rest_client:1474] ERROR - apparently rebalance progress code in infinite loop: 17.96875

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

172.23.107.145-20170623-0030-diag.zip
4.45 MB
23/Jun/17 1:30 PM
172.23.107.187-20170623-0028-diag.zip
11.33 MB
23/Jun/17 1:30 PM
172.23.107.234-20170623-0031-diag.zip
4.69 MB
23/Jun/17 1:30 PM

Issue Links

is triggering

MB-25022 MossStore created orphan files when index is created with "maxPartitionsPerPIndex": 64

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Aruna Piravi (Inactive)

Reporter:: Aruna Piravi (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 23/Jun/17 1:30 PM

Updated:: 12/Jul/17 4:08 PM

Resolved:: 06/Jul/17 11:28 AM

Gerrit Reviews

There are no open Gerrit changes

Show There are 2 closed Gerrit changes

Hide There are 2 closed Gerrit changes

MB-25015: Stop rebalance if bucket/index is removed: Gerrit Review:

MB-25015: Pass along cbgt manager to Ctl context via options: Gerrit Review:

[FTS] Rebalance-out of fts nodes stuck for 8+ mins

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty