Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.5.0
-
Untriaged
-
Unknown
Description
Build
4.5.0-2478
Testcase
./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P0 -t fts.moving_topology_fts.MovingTopFTS.swap_rebalance_kv_during_index_building,items=30000,cluster=D,F,replicas=0,GROUP=P0
This is not going to be reproducible with this testcase. I only see this failure sometimes, on various different testcases. It could very well be a case of slow disk draining because all vms are busy draining to disk but with index count not going up for >2 mins, I'm creating this bug just to rule out any issues from our side.
Steps:
1. D(.96),F(.190) cluster
2. Begin index building on F with 30K keys.
3. In parallel, swap rebalance D with another D node. [.96 --> .54]]
4. Rebalance completes successfully.
5. Index count on .190 continues to increase until 2016-05-05 23:13:58,156, then doesn't for 2 mins.
[2016-05-05 23:13:02,772] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 17263
|
[2016-05-05 23:13:08,900] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 18542
|
[2016-05-05 23:13:15,034] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 20437
|
[2016-05-05 23:13:21,188] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 21792
|
[2016-05-05 23:13:27,340] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 23446
|
[2016-05-05 23:13:33,571] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 24438
|
[2016-05-05 23:13:39,675] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 25611
|
[2016-05-05 23:13:45,878] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 26835
|
[2016-05-05 23:13:52,007] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 28057
|
[2016-05-05 23:13:58,156] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29353
|
[2016-05-05 23:14:04,256] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:10,369] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:16,483] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:22,606] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:28,748] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:34,860] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:40,978] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:47,104] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:53,221] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:14:59,328] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:05,472] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:11,576] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:17,700] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:23,821] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:29,938] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:36,059] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:42,186] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:48,297] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:15:54,418] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:16:00,534] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:16:06,671] - [fts_base:2405] INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 29786
|
[2016-05-05 23:16:12,678] - [fts_base:2425] INFO - FTS indexed 29786 docs in 3.17 mins
|
Eventually index count does reach 30000 as you can see from the last line of test log
Testlog : https://gist.github.com/arunapiravi/5acbd6f190ffe53fa95dfc65127aec9e
Attaching cbcollect info.