Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25714

[FTS] Windows: Very high memory/cpu usage

    XMLWordPrintable

Details

    Description

      Build
      5.0.0-3490

      Testcase
      ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D:F:F,index_retry=200,GROUP=MSTORE -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D,F,F,dgm_run=1,active_resident_ratio=10,moss_compact_threshold=50,GROUP=MSTORE

      This test has passed in the previous run on the same environment. Also in the past, when a node suddenly becomes unreachable via ssh, it's mostly been high memory usage.

      1. 3 nodes with kv+fts services. Each of these nodes have 16gb ram and 150gb ssd.
      2. create a default bucket of size 256mb.
      3. change fts_ram quota to 2048mb.
      4. load until active resident ratio is < 10%. 1.58M docs loaded.
      5. create default index with compaction percentage = 50%, indexing completes.
      6. when the script tries to ssh into the machine to check logs for panic, the node .33 does not respond.

      [2017-08-14 15:25:40,160] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 198
      		 
      [2017-08-14 15:25:46,638] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 42502
      		 
      [2017-08-14 15:25:52,909] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 94970
      		 
      [2017-08-14 15:25:59,575] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 141180
      		 
      [2017-08-14 15:26:06,303] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 185166
      		 
      [2017-08-14 15:26:12,752] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 218168
      		 
      [2017-08-14 15:26:18,849] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 251815
      		 
      [2017-08-14 15:26:24,977] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 282710
      		 
      [2017-08-14 15:26:31,659] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 312647
      		 
      [2017-08-14 15:26:37,776] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 333910
      		 
      [2017-08-14 15:26:44,796] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 351968
      		 
      [2017-08-14 15:26:50,894] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 370767
      		 
      [2017-08-14 15:26:57,382] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 395628
      		 
      [2017-08-14 15:27:03,471] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 413649
      		 
      [2017-08-14 15:27:09,577] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 435523
      		 
      [2017-08-14 15:27:15,772] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 456558
      		 
      [2017-08-14 15:27:21,915] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 480295
      		 
      [2017-08-14 15:27:28,004] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 499349
      		 
      [2017-08-14 15:27:34,131] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 515296
      		 
      [2017-08-14 15:27:40,229] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 530938
      		 
      [2017-08-14 15:27:46,528] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 550307
      		 
      [2017-08-14 15:27:53,098] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 579457
      		 
      [2017-08-14 15:27:59,246] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 606347
      		 
      [2017-08-14 15:28:05,864] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 633449
      		 
      [2017-08-14 15:28:11,984] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 657785
      		 
      [2017-08-14 15:28:19,299] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 683231
      		 
      [2017-08-14 15:28:26,823] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 728086
      		 
      [2017-08-14 15:28:33,219] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 772048
      		 
      [2017-08-14 15:28:39,855] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 809776
      		 
      [2017-08-14 15:28:46,891] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 851381
      		 
      [2017-08-14 15:28:53,879] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 886251
      		 
      [2017-08-14 15:29:00,011] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 920873
      		 
      [2017-08-14 15:29:06,903] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 955166
      		 
      [2017-08-14 15:29:13,667] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 993580
      		 
      [2017-08-14 15:29:20,107] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1022920
      		 
      [2017-08-14 15:29:27,892] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1061359
      		 
      [2017-08-14 15:29:40,055] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1134775
      		 
      [2017-08-14 15:29:46,670] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1165941
      		 
      [2017-08-14 15:29:52,827] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1201345
      		 
      [2017-08-14 15:29:59,955] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1232578
      		 
      [2017-08-14 15:30:06,748] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1261133
      		 
      [2017-08-14 15:30:12,861] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1287375
      		 
      [2017-08-14 15:30:19,023] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1309037
      		 
      [2017-08-14 15:30:25,167] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1323863
      		 
      [2017-08-14 15:30:31,292] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1345173
      		 
      [2017-08-14 15:30:37,406] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1360261
      		 
      [2017-08-14 15:30:43,651] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1370671
      		 
      [2017-08-14 15:30:49,801] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1381814
      		 
      [2017-08-14 15:30:55,899] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1390452
      		 
      [2017-08-14 15:31:02,000] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1400032
      		 
      [2017-08-14 15:31:08,101] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1409642
      		 
      [2017-08-14 15:31:14,197] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1417953
      		 
      [2017-08-14 15:31:20,332] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1425569
      		 
      [2017-08-14 15:31:26,433] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1438806
      		 
      [2017-08-14 15:31:32,530] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1453367
      		 
      [2017-08-14 15:31:38,650] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1470206
      		 
      [2017-08-14 15:31:44,744] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1485111
      		 
      [2017-08-14 15:31:50,860] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1495742
      		 
      [2017-08-14 15:31:56,969] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1502099
      		 
      [2017-08-14 15:32:03,070] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1507405
      		 
      [2017-08-14 15:32:09,171] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1513292
      		 
      [2017-08-14 15:32:15,267] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1519153
      		 
      [2017-08-14 15:32:21,376] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1522880
      		 
      [2017-08-14 15:32:27,473] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1528119
      		 
      [2017-08-14 15:32:33,570] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1532687
      		 
      [2017-08-14 15:32:39,666] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1536523
      		 
      [2017-08-14 15:32:45,760] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1540288
      		 
      [2017-08-14 15:32:51,857] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1544230
      		 
      [2017-08-14 15:32:57,968] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1548009
      		 
      [2017-08-14 15:33:04,105] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1551031
      		 
      [2017-08-14 15:33:10,201] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1553739
      		 
      [2017-08-14 15:33:16,300] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1557719
      		 
      [2017-08-14 15:33:22,440] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1560306
      		 
      [2017-08-14 15:33:28,539] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1563515
      		 
      [2017-08-14 15:33:34,635] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1567214
      		 
      [2017-08-14 15:33:40,731] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1569204
      		 
      [2017-08-14 15:33:46,829] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1572893
      		 
      [2017-08-14 15:33:52,926] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1574598
      		 
      [2017-08-14 15:33:59,026] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1578439
      		 
      [2017-08-14 15:34:05,128] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1579940
      		 
      [2017-08-14 15:34:11,224] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1581000
      		 
      [2017-08-14 15:34:11,304] - [fts_base:3605] INFO - Docs in index default_index_1=1581000, bucket docs=1581000
      		 
      [2017-08-14 15:34:11,325] - [rest_client:1409] INFO - /diag/eval status on 172.23.98.33:8091: True content: "c:/Program Files/Couchbase/Server/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
      		 
      [2017-08-14 15:34:11,331] - [remote_util:197] INFO - connecting to 172.23.98.33 with username:Administrator password:Membase123 ssh_key:
      		 
      [2017-08-14 15:34:26,395] - [remote_util:229] ERROR - Can't establish SSH session to node 172.23.98.33 :

      Please check logs for signs of high memory/cpu usage during indexing.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-29142 service_rebalance_failed,fts and linked_process_died failures in fts windows system test

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              apiravi Aruna Piravi (Inactive)
              apiravi Aruna Piravi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty