Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19270

[FTS] Multiple OOMs during indexing of 20K Spanish wiki docs

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 4.5.0
    • 4.5.0
    • cbft
    • Untriaged
    • Unknown

    Description

      Build
      4.5.0-2151

      Testcase
      ./testrunner -i INI_FILE.ini -p get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=P1 -t fts.stable_topology_fts.StableTopFTS.index_wiki,items=20000,lang=ES,cluster=D+F,F,F,GROUP=P1

      This used to be the same test that caused MB-19179. Now I do not see fdb crashes anymore but there are multiple OOMs preceded by conflicting metakv changes.

      I also sense this could be a duplicate of MB-19234. Hence assigning this to Marty.

      Steps
      1. Cluster: D+F,F,F
      2. Load 20K wiki docs in a bucket.
      3. Create a default index. Index starts building.
      Doc count api shows values rising and falling constantly. Then suddenly, fts rest calls time out followed by ns_server's.

      Couchbase server becomes unresponsive after sometime. This leaves the cluster in a pretty bad state.

      Apr 18 14:22:58 localhost kernel: Out of memory: Kill process 14705 (cbft) score 372 or sacrifice child
      		 
      Apr 18 14:32:54 localhost kernel: Out of memory: Kill process 23102 (cbft) score 375 or sacrifice child
      		 
      Apr 18 14:40:49 localhost kernel: Out of memory: Kill process 23257 (cbft) score 371 or sacrifice child
      		 
      Apr 18 14:45:12 localhost kernel: Out of memory: Kill process 23751 (cbft) score 360 or sacrifice child
      		 
      Apr 18 15:01:36 localhost kernel: Out of memory: Kill process 24102 (cbft) score 359 or sacrifice child
      		 
      Apr 18 15:19:37 localhost kernel: Out of memory: Kill process 24897 (cbft) score 358 or sacrifice child
      		 
      Apr 18 15:31:28 localhost kernel: Out of memory: Kill process 26119 (cbft) score 360 or sacrifice child
      		 
      Apr 18 15:50:38 localhost kernel: Out of memory: Kill process 26538 (cbft) score 406 or sacrifice child
      		 
      Apr 18 15:50:38 localhost kernel: Out of memory: Kill process 14165 (beam.smp) score 32 or sacrifice child

      Test log:

      [2016-04-18 14:09:35,926] - [rest_client:2099] INFO - Index wiki_index created
      		 
      [2016-04-18 14:09:35,926] - [fts_base:2457] INFO - Validating index distribution for wiki_index ...
      		 
      [2016-04-18 14:09:35,968] - [fts_base:2369] INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ...
      		 
      [2016-04-18 14:09:41,069] - [fts_base:2475] INFO - Validated: Number of PIndexes = 32
      		 
      [2016-04-18 14:09:41,180] - [fts_base:2486] INFO - Validated: Every pIndex serves 32 partitions or lesser
      		 
      [2016-04-18 14:09:41,180] - [fts_base:2502] INFO - Validated: pIndexes are distributed across [u'865666dc538103e38ffcdfcb01a56c06', u'a4f06c041d85658937210eafdb2f5812', u'88d32596c02f7ae26c4af0cbb0921680'] 
      [2016-04-18 14:09:41,180] - [fts_base:2508] INFO - Expecting num of partitions in each node in range 309-373
      		 
      [2016-04-18 14:09:41,181] - [fts_base:2523] INFO - Validated: Node 865666dc538103e38ffcdfcb01a56c06 houses 11 pindexes which serve 352 partitions
      		 
      [2016-04-18 14:09:41,181] - [fts_base:2523] INFO - Validated: Node a4f06c041d85658937210eafdb2f5812 houses 10 pindexes which serve 320 partitions
      		 
      [2016-04-18 14:09:41,181] - [fts_base:2523] INFO - Validated: Node 88d32596c02f7ae26c4af0cbb0921680 houses 11 pindexes which serve 352 partitions
      		 
      [2016-04-18 14:09:41,420] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 105
      		 
      [2016-04-18 14:09:47,009] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 286
      		 
      [2016-04-18 14:09:52,355] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 427
      		 
      [2016-04-18 14:09:57,801] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 758
      		 
      [2016-04-18 14:10:03,359] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 790
      		 
      [2016-04-18 14:10:09,018] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 907
      		 
      [2016-04-18 14:10:14,584] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 1118
      		 
      [2016-04-18 14:10:19,964] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 1245
      		 
      [2016-04-18 14:10:25,481] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 1597
      		 
      [2016-04-18 14:10:31,116] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 1395
      		 
      [2016-04-18 14:10:37,509] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 1729
      		 
      [2016-04-18 14:10:42,834] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 2137
      		 
      [2016-04-18 14:10:48,256] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 1766
      		 
      [2016-04-18 14:10:53,814] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 1846
      		 
      [2016-04-18 14:10:59,394] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 2119
      		 
      [2016-04-18 14:11:04,995] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 2163
      		 
      [2016-04-18 14:11:10,583] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 2779
      		 
      [2016-04-18 14:11:15,891] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 2416
      		 
      [2016-04-18 14:11:22,129] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 2566
      		 
      [2016-04-18 14:11:27,947] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 2781
      		 
      [2016-04-18 14:11:33,325] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3170
      		 
      [2016-04-18 14:11:39,279] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3547
      		 
      [2016-04-18 14:11:44,799] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3562
      		 
      [2016-04-18 14:11:50,380] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3549
      		 
      [2016-04-18 14:11:56,251] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3182
      		 
      [2016-04-18 14:12:01,548] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3515
      		 
      [2016-04-18 14:12:07,258] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 4058
      		 
      [2016-04-18 14:12:12,538] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3632
      		 
      [2016-04-18 14:12:17,850] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 4587
      		 
      [2016-04-18 14:12:23,161] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 3984
      		 
      [2016-04-18 14:12:28,438] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 4611
      		 
      [2016-04-18 14:12:34,072] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5079
      		 
      [2016-04-18 14:12:39,671] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 4278
      		 
      [2016-04-18 14:12:45,255] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5070
      		 
      [2016-04-18 14:12:50,654] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 4639
      		 
      [2016-04-18 14:12:56,131] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 4701
      		 
      [2016-04-18 14:13:01,910] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5795
      		 
      [2016-04-18 14:13:07,350] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5486
      		 
      [2016-04-18 14:13:12,996] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5097
      		 
      [2016-04-18 14:13:18,735] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5930
      		 
      [2016-04-18 14:13:24,298] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6158
      		 
      [2016-04-18 14:13:30,204] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5385
      		 
      [2016-04-18 14:13:35,484] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6389
      		 
      [2016-04-18 14:13:41,098] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6341
      		 
      [2016-04-18 14:13:46,338] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6590
      		 
      [2016-04-18 14:13:51,756] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 5987
      		 
      [2016-04-18 14:13:57,939] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6037
      		 
      [2016-04-18 14:14:03,353] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6826
      		 
      [2016-04-18 14:14:09,135] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7234
      		 
      [2016-04-18 14:14:14,394] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7425
      		 
      [2016-04-18 14:14:20,062] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7076
      		 
      [2016-04-18 14:14:25,432] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6696
      		 
      [2016-04-18 14:14:31,061] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7705
      		 
      [2016-04-18 14:14:36,719] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7630
      		 
      [2016-04-18 14:14:42,392] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 6942
      		 
      [2016-04-18 14:14:47,889] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7736
      		 
      [2016-04-18 14:14:53,209] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7197
      		 
      [2016-04-18 14:14:58,628] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 8114
      		 
      [2016-04-18 14:15:03,945] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 7385
      		 
      [2016-04-18 14:15:09,624] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 8448
      		 
      [2016-04-18 14:15:15,296] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 8537
      		 
      [2016-04-18 14:15:20,602] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 8650
      		 
      [2016-04-18 14:15:25,886] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 8939
      		 
      [2016-04-18 14:15:31,590] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 8840
      		 
      [2016-04-18 14:15:36,929] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9143
      		 
      [2016-04-18 14:15:42,069] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10318
      		 
      [2016-04-18 14:15:47,471] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10475
      		 
      [2016-04-18 14:15:52,823] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10594
      		 
      [2016-04-18 14:15:58,155] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9796
      		 
      [2016-04-18 14:16:03,621] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10068
      		 
      [2016-04-18 14:16:08,884] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10031
      		 
      [2016-04-18 14:16:14,175] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11585
      		 
      [2016-04-18 14:16:19,405] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9256
      		 
      [2016-04-18 14:16:24,562] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9467
      		 
      [2016-04-18 14:16:29,839] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9616
      		 
      [2016-04-18 14:16:35,125] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9623
      		 
      [2016-04-18 14:16:40,670] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9725
      		 
      [2016-04-18 14:16:46,814] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 9740
      		 
      [2016-04-18 14:16:52,249] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12213
      		 
      [2016-04-18 14:16:57,421] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12298
      		 
      [2016-04-18 14:17:02,720] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10047
      		 
      [2016-04-18 14:17:08,051] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12525
      		 
      [2016-04-18 14:17:13,333] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10058
      		 
      [2016-04-18 14:17:18,548] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10153
      		 
      [2016-04-18 14:17:23,797] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10183
      		 
      [2016-04-18 14:17:29,445] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13084
      		 
      [2016-04-18 14:17:34,967] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10242
      		 
      [2016-04-18 14:17:40,260] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13131
      		 
      [2016-04-18 14:17:45,548] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13469
      		 
      [2016-04-18 14:17:50,918] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13296
      		 
      [2016-04-18 14:17:56,241] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10863
      		 
      [2016-04-18 14:18:01,609] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10476
      		 
      [2016-04-18 14:18:06,799] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10624
      		 
      [2016-04-18 14:18:12,081] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12659
      		 
      [2016-04-18 14:18:17,614] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13136
      		 
      [2016-04-18 14:18:22,790] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13293
      		 
      [2016-04-18 14:18:28,273] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13182
      		 
      [2016-04-18 14:18:34,103] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10985
      		 
      [2016-04-18 14:18:39,700] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12681
      		 
      [2016-04-18 14:18:45,112] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11077
      		 
      [2016-04-18 14:18:50,670] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13050
      		 
      [2016-04-18 14:18:55,894] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12822
      		 
      [2016-04-18 14:19:01,138] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11025
      		 
      [2016-04-18 14:19:06,246] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12774
      		 
      [2016-04-18 14:19:11,641] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11212
      		 
      [2016-04-18 14:19:16,897] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12793
      		 
      [2016-04-18 14:19:22,101] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12992
      		 
      [2016-04-18 14:19:27,518] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11507
      		 
      [2016-04-18 14:19:33,204] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13170
      		 
      [2016-04-18 14:19:38,362] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13683
      		 
      [2016-04-18 14:19:43,695] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11546
      		 
      [2016-04-18 14:19:48,852] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13999
      		 
      [2016-04-18 14:19:54,263] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11622
      		 
      [2016-04-18 14:19:59,753] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13145
      		 
      [2016-04-18 14:20:04,896] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13960
      		 
      [2016-04-18 14:20:10,199] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13810
      		 
      [2016-04-18 14:20:15,325] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13965
      		 
      [2016-04-18 14:20:20,612] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13884
      		 
      [2016-04-18 14:20:25,750] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14095
      		 
      [2016-04-18 14:20:30,896] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12191
      		 
      [2016-04-18 14:20:36,144] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13119
      		 
      [2016-04-18 14:20:41,274] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13418
      		 
      [2016-04-18 14:20:46,439] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12214
      		 
      [2016-04-18 14:20:51,597] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13200
      		 
      [2016-04-18 14:20:56,832] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13359
      		 
      [2016-04-18 14:21:01,973] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13519
      		 
      [2016-04-18 14:21:07,483] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12592
      		 
      [2016-04-18 14:21:12,615] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12382
      		 
      [2016-04-18 14:21:18,032] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13748
      		 
      [2016-04-18 14:21:23,725] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14402
      		 
      [2016-04-18 14:21:28,936] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13469
      		 
      [2016-04-18 14:21:34,055] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13571
      		 
      [2016-04-18 14:21:39,187] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13377
      		 
      [2016-04-18 14:21:44,349] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14418
      		 
      [2016-04-18 14:21:49,494] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14552
      		 
      [2016-04-18 14:21:54,578] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13614
      		 
      [2016-04-18 14:21:59,782] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14379
      		 
      [2016-04-18 14:22:04,993] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13474
      		 
      [2016-04-18 14:22:10,659] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13657
      		 
      [2016-04-18 14:22:15,885] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13091
      		 
      [2016-04-18 14:22:21,123] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13711
      		 
      [2016-04-18 14:22:26,243] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14570
      		 
      [2016-04-18 14:22:31,732] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13021
      		 
      [2016-04-18 14:22:37,175] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13309
      		 
      [2016-04-18 14:22:42,412] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13281
      		 
      [2016-04-18 14:22:47,888] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13729
      		 
      [2016-04-18 14:22:53,031] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13370
      		 
      [2016-04-18 14:22:58,175] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13425
      		 
      [2016-04-18 14:23:03,298] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13829
      		 
      [2016-04-18 14:23:08,658] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13457
      		 
      [2016-04-18 14:23:14,172] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13613
      		 
      [2016-04-18 14:23:19,349] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14662
      		 
      [2016-04-18 14:23:24,761] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13849
      		 
      [2016-04-18 14:23:30,175] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14747
      		 
      [2016-04-18 14:23:35,340] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14795
      		 
      [2016-04-18 14:23:40,572] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13984
      		 
      [2016-04-18 14:23:45,791] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14829
      		 
      [2016-04-18 14:23:51,193] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14818
      		 
      [2016-04-18 14:23:56,389] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14865
      		 
      [2016-04-18 14:24:01,561] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14818
      		 
      [2016-04-18 14:24:06,737] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14013
      		 
      [2016-04-18 14:24:11,934] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14881
      		 
      [2016-04-18 14:24:17,089] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13971
      		 
      [2016-04-18 14:24:22,453] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14059
      		 
      [2016-04-18 14:24:27,591] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14987
      		 
      [2016-04-18 14:24:32,763] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14105
      		 
      [2016-04-18 14:24:37,934] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14201
      		 
      [2016-04-18 14:24:43,083] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15028
      		 
      [2016-04-18 14:24:48,437] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14190
      		 
      [2016-04-18 14:24:53,909] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14119
      		 
      [2016-04-18 14:24:59,358] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14289
      		 
      [2016-04-18 14:25:04,599] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14423
      		 
      [2016-04-18 14:25:10,246] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15081
      		 
      [2016-04-18 14:25:15,364] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14142
      		 
      [2016-04-18 14:25:20,979] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15178
      		 
      [2016-04-18 14:25:26,147] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14736
      		 
      [2016-04-18 14:25:31,399] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15217
      		 
      [2016-04-18 14:25:36,693] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15375
      		 
      [2016-04-18 14:25:41,836] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14785
      		 
      [2016-04-18 14:25:46,955] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15183
      		 
      [2016-04-18 14:25:52,374] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15390
      		 
      [2016-04-18 14:25:57,553] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15345
      		 
      [2016-04-18 14:26:02,797] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14716
      		 
      [2016-04-18 14:26:07,974] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14599
      		 
      [2016-04-18 14:26:13,308] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15417
      		 
      [2016-04-18 14:26:18,719] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14606
      		 
      [2016-04-18 14:26:24,011] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15445
      		 
      [2016-04-18 14:26:29,198] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15530
      		 
      [2016-04-18 14:26:34,416] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15242
      		 
      [2016-04-18 14:26:40,473] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15595
      		 
      [2016-04-18 14:26:49,856] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15079
      		 
      [2016-04-18 14:26:55,857] - [rest_client:760] ERROR - socket error while connecting to http://172.23.107.145:8094/api/index/wiki_index/count error [Errno 104] Connection reset by peer 
      [2016-04-18 14:26:59,205] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13045
      		 
      [2016-04-18 14:27:05,024] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14097
      		 
      [2016-04-18 14:27:10,259] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11759
      		 
      [2016-04-18 14:27:15,391] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12009
      		 
      [2016-04-18 14:27:20,744] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 10937
      		 
      [2016-04-18 14:27:25,884] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12228
      		 
      [2016-04-18 14:27:31,027] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11191
      		 
      [2016-04-18 14:27:36,337] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 11569
      		 
      [2016-04-18 14:27:41,500] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12554
      		 
      [2016-04-18 14:27:46,779] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12513
      		 
      [2016-04-18 14:27:52,043] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12438
      		 
      [2016-04-18 14:27:57,287] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14884
      		 
      [2016-04-18 14:28:02,632] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15308
      		 
      [2016-04-18 14:28:07,798] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13615
      		 
      [2016-04-18 14:28:12,966] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 12843
      		 
      [2016-04-18 14:28:18,248] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 13653
      		 
      [2016-04-18 14:28:23,546] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15444
      		 
      [2016-04-18 14:28:29,340] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15737
      		 
      [2016-04-18 14:28:34,665] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15863
      		 
      [2016-04-18 14:28:40,451] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14031
      		 
      [2016-04-18 14:28:45,793] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16292
      		 
      [2016-04-18 14:28:51,052] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15045
      		 
      [2016-04-18 14:28:56,205] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14777
      		 
      [2016-04-18 14:29:01,666] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16543
      		 
      [2016-04-18 14:29:06,815] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16359
      		 
      [2016-04-18 14:29:11,970] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15890
      		 
      [2016-04-18 14:29:17,119] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16484
      		 
      [2016-04-18 14:29:22,340] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14851
      		 
      [2016-04-18 14:29:27,571] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14853
      		 
      [2016-04-18 14:29:32,784] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16573
      		 
      [2016-04-18 14:29:37,969] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14869
      		 
      [2016-04-18 14:29:43,422] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16642
      		 
      [2016-04-18 14:29:48,587] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16098
      		 
      [2016-04-18 14:29:54,113] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14896
      		 
      [2016-04-18 14:29:59,460] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15019
      		 
      [2016-04-18 14:30:04,637] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16082
      		 
      [2016-04-18 14:30:09,811] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16816
      		 
      [2016-04-18 14:30:14,928] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15165
      		 
      [2016-04-18 14:30:20,088] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16888
      		 
      [2016-04-18 14:30:25,312] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15007
      		 
      [2016-04-18 14:30:30,587] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 17008
      		 
      [2016-04-18 14:30:35,748] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16351
      		 
      [2016-04-18 14:30:40,899] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 14894
      		 
      [2016-04-18 14:30:46,250] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15248
      		 
      [2016-04-18 14:30:51,369] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 16177
      		 
      [2016-04-18 14:30:56,618] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 17163
      		 
      [2016-04-18 14:31:01,925] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 17229
      		 
      [2016-04-18 14:31:07,176] - [fts_base:2391] INFO - Docs in bucket = 20000, docs in FTS index 'wiki_index': 15068
      		 
      [2016-04-18 14:31:42,233] - [rest_client:760] ERROR - socket error while connecting to http://172.23.107.145:8094/api/index/wiki_index/count error timed out 
      ERROR
      		 
      [2016-04-18 14:33:42,343] - [rest_client:760] ERROR - socket error while connecting to http://172.23.107.187:8091/diag/eval/ error timed out 
      ERROR

      Attaching cbcollect info.
      .145 --> kv+fts
      .234 --> fts
      .187 --> fts

      Live cluster - http://172.23.107.187:8091/ui/index.html

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            apiravi Aruna Piravi (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty