Details
-
Bug
-
Resolution: Fixed
-
Critical
-
5.0.0
-
Triaged
-
-
No
Description
Build
5.0.0-3490
Testcase
./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D:F:F,index_retry=200,GROUP=MSTORE -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D,F,F,dgm_run=1,active_resident_ratio=10,moss_compact_threshold=50,GROUP=MSTORE
This test has passed in the previous run on the same environment. Also in the past, when a node suddenly becomes unreachable via ssh, it's mostly been high memory usage.
1. 3 nodes with kv+fts services. Each of these nodes have 16gb ram and 150gb ssd.
2. create a default bucket of size 256mb.
3. change fts_ram quota to 2048mb.
4. load until active resident ratio is < 10%. 1.58M docs loaded.
5. create default index with compaction percentage = 50%, indexing completes.
6. when the script tries to ssh into the machine to check logs for panic, the node .33 does not respond.
[2017-08-14 15:25:40,160] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 198 |
[2017-08-14 15:25:46,638] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 42502 |
[2017-08-14 15:25:52,909] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 94970 |
[2017-08-14 15:25:59,575] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 141180 |
[2017-08-14 15:26:06,303] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 185166 |
[2017-08-14 15:26:12,752] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 218168 |
[2017-08-14 15:26:18,849] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 251815 |
[2017-08-14 15:26:24,977] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 282710 |
[2017-08-14 15:26:31,659] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 312647 |
[2017-08-14 15:26:37,776] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 333910 |
[2017-08-14 15:26:44,796] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 351968 |
[2017-08-14 15:26:50,894] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 370767 |
[2017-08-14 15:26:57,382] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 395628 |
[2017-08-14 15:27:03,471] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 413649 |
[2017-08-14 15:27:09,577] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 435523 |
[2017-08-14 15:27:15,772] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 456558 |
[2017-08-14 15:27:21,915] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 480295 |
[2017-08-14 15:27:28,004] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 499349 |
[2017-08-14 15:27:34,131] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 515296 |
[2017-08-14 15:27:40,229] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 530938 |
[2017-08-14 15:27:46,528] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 550307 |
[2017-08-14 15:27:53,098] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 579457 |
[2017-08-14 15:27:59,246] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 606347 |
[2017-08-14 15:28:05,864] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 633449 |
[2017-08-14 15:28:11,984] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 657785 |
[2017-08-14 15:28:19,299] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 683231 |
[2017-08-14 15:28:26,823] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 728086 |
[2017-08-14 15:28:33,219] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 772048 |
[2017-08-14 15:28:39,855] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 809776 |
[2017-08-14 15:28:46,891] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 851381 |
[2017-08-14 15:28:53,879] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 886251 |
[2017-08-14 15:29:00,011] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 920873 |
[2017-08-14 15:29:06,903] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 955166 |
[2017-08-14 15:29:13,667] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 993580 |
[2017-08-14 15:29:20,107] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1022920 |
[2017-08-14 15:29:27,892] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1061359 |
[2017-08-14 15:29:40,055] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1134775 |
[2017-08-14 15:29:46,670] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1165941 |
[2017-08-14 15:29:52,827] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1201345 |
[2017-08-14 15:29:59,955] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1232578 |
[2017-08-14 15:30:06,748] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1261133 |
[2017-08-14 15:30:12,861] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1287375 |
[2017-08-14 15:30:19,023] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1309037 |
[2017-08-14 15:30:25,167] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1323863 |
[2017-08-14 15:30:31,292] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1345173 |
[2017-08-14 15:30:37,406] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1360261 |
[2017-08-14 15:30:43,651] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1370671 |
[2017-08-14 15:30:49,801] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1381814 |
[2017-08-14 15:30:55,899] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1390452 |
[2017-08-14 15:31:02,000] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1400032 |
[2017-08-14 15:31:08,101] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1409642 |
[2017-08-14 15:31:14,197] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1417953 |
[2017-08-14 15:31:20,332] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1425569 |
[2017-08-14 15:31:26,433] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1438806 |
[2017-08-14 15:31:32,530] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1453367 |
[2017-08-14 15:31:38,650] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1470206 |
[2017-08-14 15:31:44,744] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1485111 |
[2017-08-14 15:31:50,860] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1495742 |
[2017-08-14 15:31:56,969] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1502099 |
[2017-08-14 15:32:03,070] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1507405 |
[2017-08-14 15:32:09,171] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1513292 |
[2017-08-14 15:32:15,267] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1519153 |
[2017-08-14 15:32:21,376] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1522880 |
[2017-08-14 15:32:27,473] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1528119 |
[2017-08-14 15:32:33,570] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1532687 |
[2017-08-14 15:32:39,666] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1536523 |
[2017-08-14 15:32:45,760] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1540288 |
[2017-08-14 15:32:51,857] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1544230 |
[2017-08-14 15:32:57,968] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1548009 |
[2017-08-14 15:33:04,105] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1551031 |
[2017-08-14 15:33:10,201] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1553739 |
[2017-08-14 15:33:16,300] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1557719 |
[2017-08-14 15:33:22,440] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1560306 |
[2017-08-14 15:33:28,539] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1563515 |
[2017-08-14 15:33:34,635] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1567214 |
[2017-08-14 15:33:40,731] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1569204 |
[2017-08-14 15:33:46,829] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1572893 |
[2017-08-14 15:33:52,926] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1574598 |
[2017-08-14 15:33:59,026] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1578439 |
[2017-08-14 15:34:05,128] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1579940 |
[2017-08-14 15:34:11,224] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1581000 |
[2017-08-14 15:34:11,304] - [fts_base:3605] INFO - Docs in index default_index_1=1581000, bucket docs=1581000 |
[2017-08-14 15:34:11,325] - [rest_client:1409] INFO - /diag/eval status on 172.23.98.33:8091: True content: "c:/Program Files/Couchbase/Server/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). |
[2017-08-14 15:34:11,331] - [remote_util:197] INFO - connecting to 172.23.98.33 with username:Administrator password:Membase123 ssh_key: |
[2017-08-14 15:34:26,395] - [remote_util:229] ERROR - Can't establish SSH session to node 172.23.98.33 : |
Please check logs for signs of high memory/cpu usage during indexing.
Attachments
Issue Links
- relates to
-
MB-29142 service_rebalance_failed,fts and linked_process_died failures in fts windows system test
- Closed