Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51429

Analytics service crashes and does not come up when almost 500 datasets are created

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not a Bug
    • 7.1.0
    • 7.1.0
    • analytics
    • Enterprise Edition 7.1.0 build 2475

    Description

      Steps to reproduce -

      1. Create cluster as mentioned below -
      Node Services CPU_utilization Mem_total Mem_free Swap_mem_used Active / Replica Version
      172.23.108.73 cbas 2.54279959718 3.67 GiB 2.74 GiB 0.0 Byte / 3.50 GiB 0 / 0 7.1.0-2475-enterprise
      172.23.108.63 cbas 12.213740458 3.67 GiB 2.67 GiB 0.0 Byte / 3.50 GiB 0 / 0 7.1.0-2475-enterprise
      172.23.108.76 kv 1.35883241067 3.67 GiB 3.05 GiB 0.0 Byte / 3.50 GiB 0 / 0 7.1.0-2475-enterprise
      172.23.108.64 kv, n1ql 1.69620253165 3.67 GiB 3.01 GiB 0.0 Byte / 3.50 GiB 0 / 0 7.1.0-2475-enterprise
      172.23.108.65 cbas 12.1957403651 3.67 GiB 2.71 GiB 0.0 Byte / 3.50 GiB 0 / 0 7.1.0-2475-enterprise
      172.23.108.62 cbas 12.2833843017 3.67 GiB 2.39 GiB 0.0 Byte / 3.50 GiB 0 / 0 7.1.0-2475-enterprise
      1. Create a bucket with default scope and 50 collections.
      2. Load data into collections (In this case, each collection has 2k docs)
      3. Create 500 datasets on cbas.
      4. CBAS crashes and does not come up even after an hour.
         

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          michael.blow Michael Blow added a comment -

          The analytics nodes are configured with a minimal service quota (1024 MiB), which is not designed for a production use case like 500 collections on a six node cluster.

          ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          

          michael.blow Michael Blow added a comment - The analytics nodes are configured with a minimal service quota (1024 MiB), which is not designed for a production use case like 500 collections on a six node cluster. ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.63_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.62_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.65_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: Java heap space ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded ./cbcollect_info_ns_1@172.23.108.73_20220314-141730/ns_server.analytics_info.log:Terminating due to java.lang.OutOfMemoryError: GC overhead limit exceeded
          umang.agrawal Umang added a comment -

          Looks like a testware issue. Will fix the test and rerun it to verify.

          umang.agrawal Umang added a comment - Looks like a testware issue. Will fix the test and rerun it to verify.

          People

            umang.agrawal Umang
            umang.agrawal Umang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty