Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25714

[FTS] Windows: Very high memory/cpu usage

    XMLWordPrintable

    Details

      Description

      Build
      5.0.0-3490

      Testcase
      ./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,cluster=D:F:F,index_retry=200,GROUP=MSTORE -t fts.stable_topology_fts.StableTopFTS.create_simple_default_index,cluster=D,F,F,dgm_run=1,active_resident_ratio=10,moss_compact_threshold=50,GROUP=MSTORE

      This test has passed in the previous run on the same environment. Also in the past, when a node suddenly becomes unreachable via ssh, it's mostly been high memory usage.

      1. 3 nodes with kv+fts services. Each of these nodes have 16gb ram and 150gb ssd.
      2. create a default bucket of size 256mb.
      3. change fts_ram quota to 2048mb.
      4. load until active resident ratio is < 10%. 1.58M docs loaded.
      5. create default index with compaction percentage = 50%, indexing completes.
      6. when the script tries to ssh into the machine to check logs for panic, the node .33 does not respond.

      [2017-08-14 15:25:40,160] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 198
      [2017-08-14 15:25:46,638] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 42502
      [2017-08-14 15:25:52,909] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 94970
      [2017-08-14 15:25:59,575] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 141180
      [2017-08-14 15:26:06,303] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 185166
      [2017-08-14 15:26:12,752] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 218168
      [2017-08-14 15:26:18,849] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 251815
      [2017-08-14 15:26:24,977] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 282710
      [2017-08-14 15:26:31,659] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 312647
      [2017-08-14 15:26:37,776] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 333910
      [2017-08-14 15:26:44,796] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 351968
      [2017-08-14 15:26:50,894] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 370767
      [2017-08-14 15:26:57,382] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 395628
      [2017-08-14 15:27:03,471] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 413649
      [2017-08-14 15:27:09,577] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 435523
      [2017-08-14 15:27:15,772] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 456558
      [2017-08-14 15:27:21,915] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 480295
      [2017-08-14 15:27:28,004] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 499349
      [2017-08-14 15:27:34,131] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 515296
      [2017-08-14 15:27:40,229] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 530938
      [2017-08-14 15:27:46,528] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 550307
      [2017-08-14 15:27:53,098] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 579457
      [2017-08-14 15:27:59,246] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 606347
      [2017-08-14 15:28:05,864] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 633449
      [2017-08-14 15:28:11,984] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 657785
      [2017-08-14 15:28:19,299] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 683231
      [2017-08-14 15:28:26,823] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 728086
      [2017-08-14 15:28:33,219] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 772048
      [2017-08-14 15:28:39,855] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 809776
      [2017-08-14 15:28:46,891] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 851381
      [2017-08-14 15:28:53,879] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 886251
      [2017-08-14 15:29:00,011] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 920873
      [2017-08-14 15:29:06,903] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 955166
      [2017-08-14 15:29:13,667] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 993580
      [2017-08-14 15:29:20,107] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1022920
      [2017-08-14 15:29:27,892] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1061359
      [2017-08-14 15:29:40,055] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1134775
      [2017-08-14 15:29:46,670] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1165941
      [2017-08-14 15:29:52,827] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1201345
      [2017-08-14 15:29:59,955] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1232578
      [2017-08-14 15:30:06,748] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1261133
      [2017-08-14 15:30:12,861] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1287375
      [2017-08-14 15:30:19,023] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1309037
      [2017-08-14 15:30:25,167] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1323863
      [2017-08-14 15:30:31,292] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1345173
      [2017-08-14 15:30:37,406] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1360261
      [2017-08-14 15:30:43,651] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1370671
      [2017-08-14 15:30:49,801] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1381814
      [2017-08-14 15:30:55,899] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1390452
      [2017-08-14 15:31:02,000] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1400032
      [2017-08-14 15:31:08,101] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1409642
      [2017-08-14 15:31:14,197] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1417953
      [2017-08-14 15:31:20,332] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1425569
      [2017-08-14 15:31:26,433] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1438806
      [2017-08-14 15:31:32,530] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1453367
      [2017-08-14 15:31:38,650] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1470206
      [2017-08-14 15:31:44,744] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1485111
      [2017-08-14 15:31:50,860] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1495742
      [2017-08-14 15:31:56,969] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1502099
      [2017-08-14 15:32:03,070] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1507405
      [2017-08-14 15:32:09,171] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1513292
      [2017-08-14 15:32:15,267] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1519153
      [2017-08-14 15:32:21,376] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1522880
      [2017-08-14 15:32:27,473] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1528119
      [2017-08-14 15:32:33,570] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1532687
      [2017-08-14 15:32:39,666] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1536523
      [2017-08-14 15:32:45,760] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1540288
      [2017-08-14 15:32:51,857] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1544230
      [2017-08-14 15:32:57,968] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1548009
      [2017-08-14 15:33:04,105] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1551031
      [2017-08-14 15:33:10,201] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1553739
      [2017-08-14 15:33:16,300] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1557719
      [2017-08-14 15:33:22,440] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1560306
      [2017-08-14 15:33:28,539] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1563515
      [2017-08-14 15:33:34,635] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1567214
      [2017-08-14 15:33:40,731] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1569204
      [2017-08-14 15:33:46,829] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1572893
      [2017-08-14 15:33:52,926] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1574598
      [2017-08-14 15:33:59,026] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1578439
      [2017-08-14 15:34:05,128] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1579940
      [2017-08-14 15:34:11,224] - [fts_base:3377] INFO - Docs in bucket = 1581000, docs in FTS index 'default_index_1': 1581000
      [2017-08-14 15:34:11,304] - [fts_base:3605] INFO - Docs in index default_index_1=1581000, bucket docs=1581000
      [2017-08-14 15:34:11,325] - [rest_client:1409] INFO - /diag/eval status on 172.23.98.33:8091: True content: "c:/Program Files/Couchbase/Server/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
      [2017-08-14 15:34:11,331] - [remote_util:197] INFO - connecting to 172.23.98.33 with username:Administrator password:Membase123 ssh_key:
      [2017-08-14 15:34:26,395] - [remote_util:229] ERROR - Can't establish SSH session to node 172.23.98.33 :     
      

      Please check logs for signs of high memory/cpu usage during indexing.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            steve Steve Yen added a comment -

            > (moved to alice – thanks @marty for the discussion summary)

            whoops, undo'ing the move to alice, so that we can loop in vulcan's leadership folks, including Tai Tran, Raju Suravarjjala & folks

            Show
            steve Steve Yen added a comment - > (moved to alice – thanks @marty for the discussion summary) whoops, undo'ing the move to alice, so that we can loop in vulcan's leadership folks , including Tai Tran , Raju Suravarjjala & folks
            Hide
            tai.tran Tai Tran added a comment -

            Conferred with Raju, agreed to move it to Alice.

            Show
            tai.tran Tai Tran added a comment - Conferred with Raju, agreed to move it to Alice.
            Hide
            steve Steve Yen added a comment -

            With Alice turning scorch as the default index and with Abhinav's work on enabling and improving the app-herder, this ought to be much improved – marking as resolved, and let's see what another round of testing leads to.

            Show
            steve Steve Yen added a comment - With Alice turning scorch as the default index and with Abhinav's work on enabling and improving the app-herder, this ought to be much improved – marking as resolved, and let's see what another round of testing leads to.
            Hide
            apiravi Aruna Piravi (Inactive) added a comment -

            verification blocked by CBIT-13040.

            Show
            apiravi Aruna Piravi (Inactive) added a comment - verification blocked by CBIT-13040.
            Hide
            apiravi Aruna Piravi (Inactive) added a comment -

            Verified on 6.0.0-1661. No cpu/mem issues uncovered with Scorch. Closing, thanks!

            Show
            apiravi Aruna Piravi (Inactive) added a comment - Verified on 6.0.0-1661. No cpu/mem issues uncovered with Scorch. Closing, thanks!

              People

              • Assignee:
                apiravi Aruna Piravi (Inactive)
                Reporter:
                apiravi Aruna Piravi (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.