Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62371

16c, 50G RAM, 6 FTS nodes, 100M FTS 4096 dim, 1 replica, 36 partitions/index. 100% query passing with 32% avg CPU at idle state which shoot to 100% at 1000 upserts/sec. only 11% queries could succeed.

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Critical
    • 7.6.2
    • 7.6.2
    • fts
    • Enterprise Edition 7.6.2 build 3710
    • 0

    Description

      System Config:
      16 cores, 72G(57G allotted to FTS), 6 nodes

      FTS Config
      4096 Dimensions, 100M vectors, 1 replica, 36 partitions per index

      Phase 1:
      NO kv upserts/load.
      Queries running at 2QPS
      100% queries passed
      CPU holds at 32%

      FTS Query Result Stats
      +----------------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
      | Bucket         | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
      +----------------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
      | default0_Y56JO | 6782          | count(0)       | count(6780)     | count(0)         | count(0)          | count(0)        | count(0)        |
      +----------------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
      


      Phase 2:
      1000/s kv upserts/load.
      Queries running at 2QPS
      11% queries passed
      CPU saturated to 100%

      FTS Query Result Stats
      +----------------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
      | Bucket         | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
      +----------------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
      | default0_Y56JO | 17890         | count(0)       | count(8136)     | count(9752)      | count(0)          | count(0)        | count(0)        |
      +----------------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
      

      Please note the above query stats are cumulative.

      This is a release blocker for me until we get to the right sizing wrt resources needed to serve such a workload. Please counter if you disagree.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            abhinav Abhi Dangeti
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty