Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
7.2.1
-
7.2.1(couchbase-cloud-server-7-2-1-5837-v1-0-19)
-
Untriaged
-
0
-
Unknown
Description
- Create a 11 node cluster: 3KV, 2GSI, 2N1QL, 2FTS, 2CBAS
- Create magma bucket, 2 collections, load 50M items per collection
- Create GSI, CBAS, FTS indexes and start n1ql, fts, cbas query workload
- Scale up the cluster to 4KV, 3GSI, 3N1QL, 3FTS, 3CBAS nodes. N1QL queries started timing out but few queries were successful too.
- Scale up the cluster to 5KV, 4GSI, 4N1QL, 4FTS, 4CBAS nodes. Scaling is in progress...
- N1QL queries are stuck in aborted state and no more queries are successful after a point. Every query is timing out.
From Test Logs
2023-07-12 08:14:38,707 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(721) | count(0) | count(555) | count(0) | count(0) | count(135) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:16:38,710 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(775) | count(0) | count(555) | count(0) | count(0) | count(189) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:18:38,711 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(814) | count(0) | count(555) | count(0) | count(0) | count(228) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:20:38,713 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(856) | count(0) | count(555) | count(0) | count(0) | count(270) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:22:38,716 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(903) | count(0) | count(555) | count(0) | count(0) | count(317) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:24:38,719 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(937) | count(0) | count(555) | count(0) | count(0) | count(351) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:26:38,720 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(991) | count(0) | count(555) | count(0) | count(0) | count(405) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:28:38,723 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1030) | count(0) | count(555) | count(0) | count(0) | count(444) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:30:38,724 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1072) | count(0) | count(555) | count(0) | count(0) | count(486) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:32:38,726 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1119) | count(0) | count(555) | count(0) | count(0) | count(533) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:34:38,729 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1153) | count(0) | count(555) | count(0) | count(0) | count(567) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:36:38,730 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1207) | count(0) | count(555) | count(0) | count(0) | count(621) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:38:38,733 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1246) | count(0) | count(555) | count(0) | count(0) | count(660) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:40:38,734 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1288) | count(0) | count(555) | count(0) | count(0) | count(702) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:56:38,752 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1639) | count(0) | count(555) | count(0) | count(0) | count(1053) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 08:58:38,753 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1678) | count(0) | count(555) | count(0) | count(0) | count(1092) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:00:38,756 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1720) | count(0) | count(555) | count(0) | count(0) | count(1134) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:02:38,757 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1767) | count(0) | count(555) | count(0) | count(0) | count(1181) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:04:38,759 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1801) | count(0) | count(555) | count(0) | count(0) | count(1215) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:06:38,762 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1855) | count(0) | count(555) | count(0) | count(0) | count(1269) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:08:38,763 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1894) | count(0) | count(555) | count(0) | count(0) | count(1308) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:10:38,766 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1936) | count(0) | count(555) | count(0) | count(0) | count(1350) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:12:38,767 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(1983) | count(0) | count(555) | count(0) | count(0) | count(1397) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:14:38,769 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2017) | count(0) | count(555) | count(0) | count(0) | count(1431) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:16:38,772 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2071) | count(0) | count(555) | count(0) | count(0) | count(1485) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:18:38,773 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2110) | count(0) | count(555) | count(0) | count(0) | count(1524) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:20:38,776 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2152) | count(0) | count(555) | count(0) | count(0) | count(1566) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:22:38,778 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2199) | count(0) | count(555) | count(0) | count(0) | count(1613) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:24:38,780 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2233) | count(0) | count(555) | count(0) | count(0) | count(1647) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:26:38,782 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2287) | count(0) | count(555) | count(0) | count(0) | count(1701) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:28:38,785 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2326) | count(0) | count(555) | count(0) | count(0) | count(1740) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:30:38,786 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2368) | count(0) | count(555) | count(0) | count(0) | count(1782) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
--
2023-07-12 09:32:38,789 | test | INFO | Thread-1 | [table_view:display:72] N1QL Query Statistics
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| Bucket | Total Queries | Failed Queries | Success Queries | Rejected Queries | Cancelled Queries | Timeout Queries | Errored Queries |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
| default0 | count(2415) | count(0) | count(555) | count(0) | count(0) | count(1829) | count(0) |
+----------+---------------+----------------+-----------------+------------------+-------------------+-----------------+-----------------+
ritesh.agarwal@MBP-2 TAF %
QE Test |
git fetch https://review.couchbase.org/TAF refs/changes/09/193609/6
|
sudo guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/couchbase_capella_volume_2.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.hostedHospital.Murphy.test_rebalance,nodes_init=6,graceful=True,skip_cleanup=false,num_items=50000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=2,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,maxttl=10,pc=20,gsi_nodes=2,cbas_nodes=2,fts_nodes=2,kv_nodes=3,n1ql_nodes=2,kv_disk=500,n1ql_disk=50,gsi_disk=500,fts_disk=500,cbas_disk=500,kv_compute=n2-standard-4,gsi_compute=n2-standard-4,n1ql_compute=n2-standard-4,fts_compute=n2-standard-4,cbas_compute=n2-standard-4,mutation_perc=20,key_type=CircularKey,capella_run=true,services=data-query-index-analytics-search,max_rebl_nodes=27,provider=GCP,region=us-east1,type=PD-SSD,size=500,skip_teardown_cleanup=true,wait_timeout=14400,index_timeout=28800,runtype=dedicated,horizontal_scale=1,skip_init=false,sanity=True -m rest'
|