Ejection makes no progress for several minutes causing TMP OOM failures during data ingestion

Several tests indicate problems with initial data load. Clients keeps receiving ep_tmp_oom_errors even in absence of large persistence and replication queues. It looks like kv-engine cannot evict items promptly - ep_num_value_ejects counter literally freezes for several minutes.

Let's use the following test case as an example:

• 2 nodes
• 1 bucket (full ejection)
• 100M items

I stopped the clients after the first TMP OOM error at 15:06:50 and left the system running.

I can see that one of three non-IO threads is constantly busy (100% CPU) and ep_num_eject_failures counter keeps increasing. Once in a while items get ejected.

Logs, perf profile for non-IO thread, and some graphs from mortimer are attached.