Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.5.0, 5.0.0
-
Untriaged
-
Unknown
-
ForestDB: Oct 17 - Nov 4
Description
Build
4.7.0-1044
Testcase
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,GROUP=FDB -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D+F,F,D+F,dgm_run=1,active_resident_ratio=10,eviction_policy=fullEviction,kvstore=forestdb,fdb_compact_interval=300,fdb_compact_threshold=20,GROUP=FDB
1. I see signs of segmentation fault from cbft in the logs. I do not see any fdb coredumps generated though.
On .120 -
2016-08-31T00:15:46.840-07:00 [INFO] stats: {
|
"default:default_index_1:avg_queries_latency": 0,
|
"default:default_index_1:batch_merge_count": 5.152852e+06,
|
"default:default_index_1:doc_count": 773827,
|
"default:default_index_1:iterator_next_count": 684986,
|
"default:default_index_1:iterator_seek_count": 0,
|
"default:default_index_1:num_bytes_live_data": 1.2996987911e+10,
|
"default:default_index_1:num_bytes_used_disk": 2.6308780032e+10,
|
"default:default_index_1:num_mutations_to_index": 232685,
|
"default:default_index_1:num_pindexes": 10,
|
"default:default_index_1:num_pindexes_actual": 10,
|
"default:default_index_1:num_pindexes_target": 10,
|
"default:default_index_1:num_recs_to_persist": 7.741526e+06,
|
"default:default_index_1:reader_get_count": 90156,
|
"default:default_index_1:reader_multi_get_count": 0,
|
"default:default_index_1:reader_prefix_iterator_count": 20,
|
"default:default_index_1:reader_range_iterator_count": 0,
|
"default:default_index_1:timer_batch_store_count": 0,
|
"default:default_index_1:timer_data_delete_count": 0,
|
"default:default_index_1:timer_data_update_count": 94054,
|
"default:default_index_1:timer_opaque_get_count": 728,
|
"default:default_index_1:timer_opaque_set_count": 408,
|
"default:default_index_1:timer_rollback_count": 0,
|
"default:default_index_1:timer_snapshot_start_count": 148,
|
"default:default_index_1:total_bytes_indexed": 2.7910156e+07,
|
"default:default_index_1:total_bytes_query_results": 0,
|
"default:default_index_1:total_compactions": 0,
|
"default:default_index_1:total_queries": 0,
|
"default:default_index_1:total_queries_error": 0,
|
"default:default_index_1:total_queries_slow": 0,
|
"default:default_index_1:total_queries_timeout": 0,
|
"default:default_index_1:total_request_time": 0,
|
"default:default_index_1:total_term_searchers": 0,
|
"default:default_index_1:writer_execute_batch_count": 601,
|
"num_bytes_used_ram": 1038537224,
|
"pct_cpu_gc": 0.007068731334610401,
|
"total_gc": 742
|
}
|
2016-08-31T00:15:48.972-07:00 [INFO] moss_herder: persistence progess, waiting: 2
|
2016-08-31T00:16:07.593-07:00 [INFO] cbdatasource: server: 172.23.106.120:11210, uprOpenName: fts:default_index_1_4ad546af6d515fb2-93d3320, worker, looping beg, vbucketState: "running" (has 160 vbuckets), 576-607, 736-799, 896-959
|
2016-08-31T00:16:07.603-07:00 [INFO] cbdatasource: server: 172.23.105.224:11210, uprOpenName: fts:default_index_1_4ad546af6d515fb2-93d3320, worker, looping beg, vbucketState: "running" (has 160 vbuckets), 32-63, 128-159, 256-287, 384-447
|
2016-08-31T00:16:27.009-07:00 [INFO] moss_herder: persistence progess, waiting: 2
|
2016-08-31T00:16:36.075-07:00 [INFO] moss_herder: persistence progess, waiting: 2
|
2016-08-31T00:16:37.598-07:00 [INFO] cbdatasource: server: 172.23.106.120:11210, uprOpenName: fts:default_index_1_4ad546af6d515fb2-93d3320, worker, looping beg, vbucketState: "running" (has 160 vbuckets), 576-607, 736-799, 896-959
|
2016-08-31T00:16:37.635-07:00 [INFO] cbdatasource: server: 172.23.105.224:11210, uprOpenName: fts:default_index_1_4ad546af6d515fb2-93d3320, worker, looping beg, vbucketState: "running" (has 160 vbuckets), 32-63, 128-159, 256-287, 384-447
|
2016-08-31T00:16:37.856-07:00 [INFO] moss_herder: persistence progess, waiting: 2
|
[goport] 2016/08/31 00:16:45 /opt/couchbase/bin/cbft terminated: signal: segmentation fault (core dumped)
|
2016-08-31T00:16:45.844-07:00 [INFO] main: /opt/couchbase/bin/cbft started (v0.3.1/4.1.0)
|
2. Initially I thought the nodes were just slow due to overloading but it appears just 1 doc has been left unindexed for over 2 mins -
[2016-08-31 00:54:52,511] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3207227
|
[2016-08-31 00:54:58,599] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3207227
|
[2016-08-31 00:55:04,970] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3209018
|
[2016-08-31 00:55:11,467] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3216196
|
[2016-08-31 00:55:17,746] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3219423
|
[2016-08-31 00:55:23,959] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:55:30,044] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:55:36,128] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:55:42,210] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:55:48,349] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:55:54,499] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:00,608] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:06,691] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:12,778] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:18,864] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:24,971] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:31,064] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:37,252] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:43,444] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:49,716] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:56:55,970] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:57:02,115] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:57:08,314] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:57:14,418] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:57:20,562] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:57:26,778] - [fts_base:2582] INFO - Docs in bucket = 3221000, docs in FTS index 'default_index_1': 3220999
|
[2016-08-31 00:57:32,785] - [fts_base:2602] INFO - FTS indexed 3220999 docs in 97.2 mins
|
[2016-08-31 00:57:32,952] - [fts_base:2808] INFO - Docs in index default_index_1=3220999, bucket docs=3221000
|
Attaching cbcollect.