Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60851

K=100 Queries causing 40+GB used on single node and 200+GB used over all 5 node, which leading to OOM after 8 queries or so

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.6.2
    • 7.6.0
    • fts
    • Untriaged
    • 0
    • No

    Description

      Test environment:

      • 5 dedicated fts nodes with 16cpus and 64GiB RAM each node
      • 60Million document on KV
      • 60Million documents have vectors of dimension 1536

      Test Steps:

      • Loaded 60Mil documents on to KV
      • Created indexing with 18 partitions per node, so overall 90 paritions
      • I have only one index with l2_norm 
      • Indexing took 1hour 17mins for 60Million documents without any OOM but memory almost went up to 99% on few nodes( One observation is , initial 40Million took only 25mins or so next 20million to complete took 
      • Left the cluster idle for nearly 10+ hours to see usage comes down, but never came down below 37% or so. For whole 10 hours per node 22+GiB is in use.
      •  
      • Now started running queries with K=1 and k=100 and then K=200 in sequential manner, after running 8 queries or so on almost on all the nodes I seen 95+% cpu usage along with one node crashed saying OOM(Node-009)
      • Queries which are successful took varying amounts of time like 25sec to 60sec.

       

      Below logs will will help understanding the behaviour using promtimer. Also attached images below which confirms the behaviour

       

      Logs:

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ashokkumar.alluri Ashok Alluri
            ashokkumar.alluri Ashok Alluri
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty