[System Test] 2 OOM killings observed in query nodes

Build : 7.0.0-4707
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml (Mad-hatter longevity script without collections)
Scale : 3
Iteration : 1st & 3rd

172.23.96.251 and 172.23.96.252 are the query nodes in this cluster. Both nodes have experienced OOM killings are different times.

172.23.96.252 experienced at Mar 16 22:04:05 2021 :

[Tue Mar 16 22:04:05 2021] Out of memory: Kill process 4869 (cbq-engine) score 892 or sacrifice child

[Tue Mar 16 22:04:05 2021] Killed process 4869 (cbq-engine) total-vm:26389968kB, anon-rss:21846196kB, file-rss:932kB, shmem-rss:0kB

172.23.96.251 experienced at Mar 17 08:21:55 2021 :

[Wed Mar 17 08:21:55 2021] Out of memory: Kill process 22527 (cbq-engine) score 864 or sacrifice child

[Wed Mar 17 08:21:55 2021] Killed process 22527 (cbq-engine) total-vm:26344784kB, anon-rss:21146412kB, file-rss:0kB, shmem-rss:0kB

Before the OOM killing on 172.23.96.251, the following lines were seen in the logs -

2021-03-17T08:21:56.475-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 

2021-03-17T08:21:56.486-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 

2021-03-17T08:21:56.488-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address

Logs for the time around OOM killing on 172.23.96.251 :
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.25.zip

Logs for the time around OOM killing on 172.23.96.252 :
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.97.242.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615961349/collectinfo-2021-03-17T060911-ns_1%40172.23.99.25.zip