Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56982

N1QL queries are failing as KV lookup couldn't finish in ~75s.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 7.1.4, 7.2.0
    • couchbase-bucket
    • 7.1.4-3632

    Description

      1. Create a 6 nodes cluster: 3KV, 3 N1ql/GSI
      2. Create a bucket, 2 collections, load 1.5b items in each.
      3. Build GSI indexes. Wait for them to be ready
      4. Start a query load asynchronously. 200 threads running queries in a loop. In each iteration query is chosen randomly from 5 queries.
      5. Start KV r:w mutations at 10K/s, 50-50%
      6. Change the disk which results in swap rebalance.
      7. 3 KV nodes swap rebalance finished in 7.5 hours
      8. 3 GSI node swap rebalance finished in 28.5 hours
      9. During the last GSI/N1QL node swap rebalance max number of queries are timing out.
      10. In UI, I don't see any data available in stats between 10pm May 16th to 6am - May 17th. That was the time i saw max queries timing out.

      cc: Dave Rigby

      Attachments

        1. image-2023-05-17-11-13-26-858.png
          image-2023-05-17-11-13-26-858.png
          380 kB
        2. image-2023-05-17-11-13-32-191.png
          image-2023-05-17-11-13-32-191.png
          409 kB
        3. Last4000CompletedRequests.json
          7.88 MB
        4. screenshot-1.png
          screenshot-1.png
          353 kB
        5. screenshot-2.png
          screenshot-2.png
          412 kB
        6. Screenshot 2023-05-22 at 16.30.49.png
          Screenshot 2023-05-22 at 16.30.49.png
          661 kB
        7. timeout.txt
          163 kB

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-57400 ActiveStream::getItemsRemaining when many items are held in checkpoints can cause KV gridlock.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MB-57296 BucketManager::forEach should not require exclusive access to the buckets

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MB-56891 Don't block updates to DcpConnmap while "dcp" and "dcpagg" stats are executing

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-56737 bucket doc stats not correct

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty