Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58090

[SizingIssue - c5.2xlarge, gsi-n1ql colocated]: Unbounded queries are failing either due to index scan timeout or due to temp file size exceeded limit.

    XMLWordPrintable

Details

    Description

      1. Create a 5 nodes(c5.2xlarge) cluster: 3-KV, 2-GSI & N1QL colocated.
      2. Create bucket, 10 collections, 100M items in each.
      3. Create GSI indexes on 2 collections.
      4. Start KV Read+expiry load at 10k ops/s(5k Reads, 5k Expiry). Start the n1ql query load in parallel.
      5. Scale up cluster to 4-KV, 3-GSI & N1QL. Finished in 16122.9400001 seconds
      6. Scale up cluster to 5-KV, 4-GSI & N1QL. Finished in 13892.306 seconds
      7. Scale down cluster to 4-KV, 3-GSI & N1QL. Rebalance still running since 7 hours.
      8. During the rebalancing + incremental index building + plasma compaction(keys getting expired in 10s) + Query workload - N1QL queries(QueryScanConsistency.NOT_BOUNDED) are timing out due to index scan couldn't complete in 75s.

      cc: Deepkaran Salooja

      QE Test

      sudo guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/couchbase_capella_volume_2_new.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.hostedHospital.Murphy.test_rebalance,num_items=100000000,num_buckets=1,bucket_names=GleamBook,bucket_type=membase,iterations=2,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,maxttl=10,pc=20,gsi_nodes=2,cbas_nodes=2,fts_nodes=2,kv_nodes=3,n1ql_nodes=2,kv_disk=1000,n1ql_disk=50,gsi_disk=500,fts_disk=1000,cbas_disk=1000,kv_compute=c5.2xlarge,gsi_compute=c5.2xlarge,n1ql_compute=c5.2xlarge,fts_compute=c5.2xlarge,cbas_compute=c5.2xlarge,mutation_perc=20,key_type=CircularKey,capella_run=true,services=data-index:query,rebl_services=data-index:query,max_rebl_nodes=27,provider=AWS,region=us-east-1,type=GP3,size=1000,collections=10,ops_rate=100000,skip_teardown_cleanup=true,wait_timeout=14400,index_timeout=28800,runtype=dedicated,skip_init=false,rebl_ops_rate=10000,collections=10,expiry=true,vh_scaling=true,horizontal_scale=1 -m rest'
      

      Attachments

        1. consoleText.txt
          11.22 MB
        2. index_scan.json
          350 kB
        3. indexScanDetails.json
          8 kB
        4. Screenshot 2023-08-04 at 12.40.40 PM.png
          Screenshot 2023-08-04 at 12.40.40 PM.png
          2.33 MB
        5. Screenshot 2023-08-04 at 12.42.06 PM.png
          Screenshot 2023-08-04 at 12.42.06 PM.png
          1.07 MB
        6. Screenshot 2023-08-04 at 12.42.44 PM.png
          Screenshot 2023-08-04 at 12.42.44 PM.png
          1.52 MB
        7. Screenshot 2023-08-04 at 12.47.13 PM.png
          Screenshot 2023-08-04 at 12.47.13 PM.png
          1.20 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            ritesh.agarwal Ritesh Agarwal
            ritesh.agarwal Ritesh Agarwal
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty