Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46206

High memory usage by indexer even when there are no indexes

    XMLWordPrintable

Details

    Description

      Summary:
      The RAM usage on indexer node (172.23.106.238) is almost 12GiB even when there are no gsi indexes. I had many indexes on collections, and then I dropped all collections (and hence all indexes got dropped). I waited close to 24 hours and the RAM usage still didn't drop. 
      Here's the RAM usage on indexer node after waiting for 24 hours after all indexes got dropped:

         PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                           
       13304 couchba+  20   0   15.8g  12.4g  19168 S   2.7 52.9 424:39.43 indexer                                                                           
      106880 couchba+  20   0 4743884 227704   3692 S   2.0  0.9 322:27.12 beam.smp                                                                          
      107810 couchba+  20   0 1280224 239304  13836 S   1.3  1.0  23:52.86 prometheus 

      Steps to Reproduce:
      1. Create 2kv, 1 n1ql, 14 index nodes cluster
      2. Create a bucket with 1K collections, and a total of 1K items
      3. Create 10 indexes per collection (so a total of 10K indexes that are deferred)
      4. Build 6K indexes from step 3
      5. Flush bucket, and drop all collections. So total index count is 0 here.
      6. Rebalance-out all index nodes except one index node (172.23.106.238)
      Wait for sufficiently long and observe that RAM usage by indexer on the above node doesn't drop as mentioned in the summary above

      Attaching logs from the resulting 4 node cluster at step 6

      Attachments

        1. c4000d950.svg
          18 kB
        2. go-a-1621344380.svg
          68 kB
        3. servers.png
          servers.png
          481 kB
        4. UI_logs.png
          UI_logs.png
          497 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            akhil.mundroy Akhil Mundroy added a comment - Sumedh Basarkod , please rerun the test with this toy build with the tentative fix: http://latestbuilds.service.couchbase.com/builds/latestbuilds/couchbase-server/toybuilds/12975/couchbase-server-enterprise-7.0.0-12975-centos7.x86_64.rpm

            Here's the logs link for the run from the above toybuild 12975 which had the tentative fix:
            http://supportal.couchbase.com/snapshot/5ac520c33b8e4a7f5603a332f6d85f29::0
            s3://cb-customers-secure/mb-46206-toybuild-12975_1/2021-05-19/prof_folder.zip
            s3://cb-customers-secure/mb-46206-toybuild-12975_1/2021-05-19/svg_folder.zip

            Did not hit into any memory issues with this. It created 10K deferred indexes successfully (which was previously failing midway because of memory leak). (But however, as a side note, the test failed because of MB-45990 when it was building those deferred indexes (so ~6K indexes were online in total))

            sumedh.basarkod Sumedh Basarkod (Inactive) added a comment - - edited Here's the logs link for the run from the above toybuild 12975 which had the tentative fix: http://supportal.couchbase.com/snapshot/5ac520c33b8e4a7f5603a332f6d85f29::0 s3://cb-customers-secure/mb-46206-toybuild-12975_1/2021-05-19/prof_folder.zip s3://cb-customers-secure/mb-46206-toybuild-12975_1/2021-05-19/svg_folder.zip Did not hit into any memory issues with this. It created 10K deferred indexes successfully (which was previously failing midway because of memory leak). (But however, as a side note, the test failed because of MB-45990 when it was building those deferred indexes (so ~6K indexes were online in total))

            Build couchbase-server-7.0.0-5191 contains gometa commit 265910a with commit message:
            MB-46206: Cleanup doc in RepoIterator.Next

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5191 contains gometa commit 265910a with commit message: MB-46206 : Cleanup doc in RepoIterator.Next

            Build couchbase-server-7.0.0-5191 contains indexing commit 6641f77 with commit message:
            MB-46206: Add Close without pool for fdb doc

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5191 contains indexing commit 6641f77 with commit message: MB-46206 : Add Close without pool for fdb doc

            Verified on 7.0.0-5219.  Closing

            sumedh.basarkod Sumedh Basarkod (Inactive) added a comment - Verified on 7.0.0-5219.  Closing

            People

              srinath.duvuru Srinath Duvuru
              sumedh.basarkod Sumedh Basarkod (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty