on_update_failure seen in 6.6.3 runs for timer tests

On resizing a timer noop performance test to be 100% resident on all three source, metadata, and destination buckets, we see the following LCB_ETIMEDOUT - on 6.6.3 runs of the same test. As we resized the previously DGM test to fit completely in memory, we can safely assume that the RR is not causing the timeouts in the test. 

Resized Test Run : http://perf.jenkins.couchbase.com/job/themis/10918/consoleFull 

Cbmonitor : http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_663-9757_process_timer_events_97e1  - Attached a chart for RR on source bkt

Original Test Run : http://perf.jenkins.couchbase.com/job/themis/10773/consoleFull 

Cbmonitor : http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=themis_663-9744_process_timer_events_3c87 

Please let us know if this is still a sizing issue. Also, the test does not do any bucket op (read or write to source or destination buckets) in the timer callback function , it is noop test. Ideally, we must not be seeing KV timeouts. Please correct me if I'm wrong.

Eventing log 

2021-06-18T05:08:17.898-07:00 [Info] eventing-consumer [worker_perf-test1_14:/tmp/127.0.0.1:8091_14_1135348043.sock:100888] [lcb,server L:646 I:3908674612] Failing command with error LCB_ETIMEDOUT (0x17): {"b":"eventing","i":"00000000e8f9a434/5e06e315d7db9bfc/106f43","l":"172.23.97.177:53098","r":"172.23.96.16:11210","s":"kv:get_cluster_config","t":2500000}

on_update_failure stats from test log : 

18:32:32 "on_delete_failure": 0, 18:32:32 "on_delete_success": 0, 18:32:32 "on_update_failure": 7, 18:32:32 "on_update_success": 99999993,