Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30456

FTS isn't responding under load with fuzzy2 queries

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 6.0.0
    • 6.0.0
    • fts
    • Cluster: atlas_setupA
      OS: CentOS 7
      CPU: E5-2680 v3 (48 vCPU)
      Memory: 256 GB
      Disk: Samsung PM863
    • Untriaged
    • Unknown

    Description

      Setup and test:
      CB build: 6.0.0-1242
      3 nodes, data+search on each node
      1M docs, Scorch
      30 client threads executing fuzzy2 queries

       

      Once load started the cbft stops responding and requests start timing out. As soon as load level goes down the cbft gets back.

      The disk utilization is 95+% but it doesn't look like disk saturation is causing the timeouts. The data service works ok also server is pretty responsive when connecting via ssh. Also disk utilization is pretty high with other heavy queries like fuzzy1 or wild.

      On the other hand,  amount of files created by cbft grows way faster with fuzzy2:

      This is either the reason or the outcome of the issue.

       

       Full comparison:

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=fts_600-1316_run_test_bd72&label=fuzzy2&snapshot=fts_600-1316_run_test_8849&label=fuzzy1&snapshot=fts_600-1316_run_test_66f0&label=wildcart&snapshot=fts_600-1316_run_test_ebab&label=med-term

       

      Logs:

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-2683/172.23.99.211.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-2683/172.23.99.39.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-2683/172.23.99.40.zip

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            steve Steve Yen
            oleksandr.gyryk Alex Gyryk (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty