John Liang added a comment - 14/Jan/22 9:18 AM Saptarshi Sen Is there any expiry or delete during query? Thanks.

Saptarshi Sen added a comment - 14/Jan/22 9:56 AM - edited From the spec in the console log, ttl is 0. So the workload should not be related to expiry.   15:28:05 "n1ql_op": "read",15:28:05 "n1ql_queries": [15:28:05 {15:28:05 "args": "[\"\{gender}\", \"\{marital_status}\", \"\{education_status}\"]",15:28:05 "scan_consistency": "not_bounded",15:28:05 "statement": "SELECT item.id, AVG(quantity) AS avg_quantity, AVG(list_price) AS avg_list_price, AVG(coupon_amt) AS avg_coupon_amt, AVG(sales_price) AS avg_sales_price FROM `bucket-1` WHERE customer.gender = $1 AND customer.marital_status = $2 AND customer.education_status = $3 GROUP BY item.id ORDER BY item.id LIMIT 100;"15:28:05 }15:28:05 ], 15:28:05 "transactionsenabled": 0, 15:28:05 "transactionupdateproportion": 0.75, ...   *15:28:05* "ttl": 0, 15:28:05 "updates": 0, {*}1{*}5:28:05 "use_backup": false, 15:28:05 "user_mod_throughput": Infinity, 15:28:05 "user_mod_workers": 0, I took a look at the logs for indexer node 172.23.97.19. From Plasma Stats, from count & items_count stat, it looks like there are only inserts "MainStore": { "memory_quota": 283115520000, "count": 100000000, .... "items_count": 100000000, "total_records": 100000000, ... "resident_ratio": 1.00000, "compressed_percent": 71, "compressed_compressible_percent":99, "mvcc_purge_ratio": 1.00000, "reader_purge_cnt": 0, "reader_emptypage_cnt": 0, ... "hole_cleaner_status": "", } Based on "reader_emptypage_count", the count for empty page is zero, and hole cleaner does not seem to run for the index. 2022-01-11T15:41:03.837-08:00 [Info] Periodic Aggregated StorageStats: { ... "reader_purge_count": 0, "reader_emptypage_count": 0 }

Srinath Duvuru added a comment - 24/Jan/22 1:37 PM Korrigan Clark Looks the performance is back up. There is some fluctuation in this test. Saptarshi Sen It does not look like a storage issue, can you confirm? Thanks.

Saptarshi Sen added a comment - 24/Jan/22 2:02 PM - edited I reverted all the commits related to holeCleaner and ran the test with a toy build (7.1.0-14332) a) https://review.couchbase.org/c/plasma/+/169280 [plasma] b) https://review.couchbase.org/c/plasma/+/169281 [plasma] c https://review.couchbase.org/c/plasma/+/169293 [plasma] d) https://review.couchbase.org/c/indexing/+/169269 [indexer config] Toy Build Parameters: http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/4332/parameters/ Perf Test : http://perf.jenkins.couchbase.com/job/arke/9862/console I still observed a similar query/sec number (1637) in the test as to what was reported for the build 2007 (1647). So it does not seem to be related to the holeCleaner changes. Reference Notes: build 2007 (indexer stats) "bucket-1:ag6:timings/n1ql_expr_eval":"988095 0 0","bucket-1:ag6:num_requests_range":0,"bucket-1:ag6:num_completed_requests_range":0,"bucket-1:ag6:num_rows_returned_range":0,"bucket-1:ag6:num_rows_scanned_range":0,"bucket-1:ag6:scan_cache_hit_range":0,"bucket-1:ag6:num_requests_aggr":988095,"bucket-1:ag6:num_completed_requests_aggr":988095,"bucket-1:ag6:num_rows_returned_aggr":98809500,"bucket-1:ag6:num_rows_scanned_aggr":1388228644,"bucket-1:ag6:scan_cache_hit_aggr":0,"10857409829870475256:completion_progress":4636737291354636288, build 2006 (indexer stats) "bucket-1:ag6:timings/n1ql_expr_eval":"1236532 0 0","bucket-1:ag6:num_requests_range":0,"bucket-1:ag6:num_completed_requests_range":0,"bucket-1:ag6:num_rows_returned_range":0,"bucket-1:ag6:num_rows_scanned_range":0,"bucket-1:ag6:scan_cache_hit_range":0,"bucket-1:ag6:num_requests_aggr":1236532,"bucket-1:ag6:num_completed_requests_aggr":1236532,"bucket-1:ag6:num_rows_returned_aggr":123653200,"bucket-1:ag6:num_rows_scanned_aggr":1737315043,"bucket-1:ag6:scan_cache_hit_aggr":0,"86388607918421446:completion_progress":4636737291354636288,

John Liang added a comment - 25/Jan/22 3:59 PM Jeelan Poola Assign back to you for now since the same regression happens even after we comment out the hole cleaning code.

John Liang added a comment - 25/Jan/22 4:33 PM - edited Btw, the latest AG6 throughput is back to 2562 in build 7.1.0-2123. Korrigan Clark It may be a good idea to do a couple of runs with build 2123 to confirm the result.

Korrigan Clark (Inactive) added a comment - 26/Jan/22 6:21 AM Ive kicked off two confirmation runs

Korrigan Clark (Inactive) added a comment - 27/Jan/22 7:42 AM runs look good... closing