Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62051

FTS OOM - Service 'fts' exited with status 137

    XMLWordPrintable

Details

    Description

      Steps:

      1. FTS Config: 
        5 FTS Nodes
        32GB ram, 16Cores cpu
      2. Load 100M vector data
      3. Create 1 index
      4. Run queries and wait for indexing

      Seeing. a lot of OOM kills on node 009. On seeing the memory profile with fts_mprof.log I don't see anything alarming. I also don't see full memory distribution in the mem graph.

       

      3:25:45 PM 24 May, 2024 
      Service 'fts' exited with status 137. Restarting. Messages:
      		 
      2024-05-24T15:25:04.861+00:00 [INFO] app_herder: indexing over indexQuota: 16660633600, memUsed: 18088530240, preIndexingMemory: 11201600, indexes: 3, waiting: 8
      		 
      2024-05-24T15:25:05.014+00:00 [INFO] app_herder: indexing over indexQuota: 16660633600, memUsed: 18089929440, preIndexingMemory: 12600800, indexes: 3, waiting: 9
      		 
      2024-05-24T15:25:06.911+00:00 [INFO] app_herder: indexing over indexQuota: 16660633600, memUsed: 19269663328, preIndexingMemory: 12600800, indexes: 3, waiting: 9
      		 
      2024-05-24T15:25:09.349+00:00 [INFO] app_herder: indexing over indexQuota: 16660633600, memUsed: 19023191928, preIndexingMemory: 12600800, indexes: 3, waiting: 9
      		 
      2024-05-24T15:25:23.344+00:00 [INFO] app_herder: indexing over indexQuota: 16660633600, memUsed: 19327774152, preIndexingMemory: 12600800, indexes: 3, waiting: 9
      		 
      2024-05-24T15:25:31.742+00:00 [INFO] app_herder: indexing over indexQuota: 16660633600, memUsed: 29843367440, preIndexingMemory: 12600800, indexes: 3, waiting: 9
      		 
      2024-05-24T15:25:34.344+00:00 [INFO] app_herder: indexing over indexQuota: 16660633600, memUsed: 30714970088, preIndexingMemory: 12600800, indexes: 3, waiting: 9
      		 
      hidens_log 000ns_1@svc-s-node-009.opckjum75pmbcubv.customsubdomain.nonprod-project-avengers.com

       

      2:31:08 PM 24 May, 2024 
       
      		 
      Service 'fts' exited with status 137. Restarting. Messages:
      		 
      2024-05-24T14:13:59.549+00:00 [INFO] feed_dcp_gocbcore: Start, name: bucket_sift_bucket_idx_sdbge-vector_data-sname_5cc83eb1212db71b_43716852, num streams: 85, manifestUID: 5, streamOptions: {FilterOptions: &{ScopeID:0 CollectionIDs:[11]}, StreamOptions: &{StreamID:2}}, vbuckets: [854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938]
      		 
      2024-05-24T14:14:06.315+00:00 [INFO] ctl: debounceCfgEvents duration: 10500 MS
      		 
      2024-05-24T14:14:06.315+00:00 [INFO] ctl: kickIndexDefs, kind: cfgEvent
      		 
      2024-05-24T14:14:06.323+00:00 [INFO] ctl: run, kind: nodeDefs-wanted, updated memberNodes: {1dc13b3909b3a3d3595b5ef079aea950;3c838e19ef22b874027af5e5534a11a7;844b1a47a34139745730b7f7deb69413;8e781469d2d09cabc572a21e893cb4e9;a36224e720efc7f2091babed2d34c009;}

       

      Attachments

        1. bytes_used_disk.png
          bytes_used_disk.png
          218 kB
        2. fts_ram.png
          fts_ram.png
          258 kB
        3. image-2024-05-25-01-10-27-514.png
          image-2024-05-25-01-10-27-514.png
          204 kB
        4. image-2024-05-25-14-49-24-468.png
          image-2024-05-25-14-49-24-468.png
          512 kB
        5. index_size_diff.png
          index_size_diff.png
          271 kB
        6. merge_ops.png
          merge_ops.png
          200 kB
        7. query_workload.png
          query_workload.png
          196 kB

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-62057 Vector Search: FTS crash looping with exit status 1 on system-test cluster

          • Test Blocker - QE-related tests can't proceed
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-62050 Underutilisation of some fts nodes in the cluster

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-61817 FTS process is oom killed during rebalance in operation.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-62058 Vector Search: FTS crashed with exit status 130

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sarthak.dua Sarthak Dua
              sarthak.dua Sarthak Dua
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty