Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
For the issues reported in MB-45046, on 172.23.96.251, between 2021-03-17T06:55:44 & 2021-03-17T08:21:56, which is a duration of ~1.5 hours, there are 1.5M+ occurences of lines like these in the query logs -
2021-03-17T08:21:56.475-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:56.486-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:56.488-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
This is a supportability issue as the logs get flooded by these lines and roll over quickly. It could be seen that within the millisecond, there could be as many as 12 duplicate lines logged.
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address
|
These can be de-duplicated to avoid quick rollover of logs and flooding them with same error messages.
Some other such lines are -
2021-03-17T06:55:45.687-07:00 [Info] GsiClient::UpdateUsecjson: using collatejson as data format between indexer and GsiClient -> 4K+ occurences in ~2 hrs
2021-03-17T06:55:45.610-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. -> 3K+ occurences
Logs :
Logs for the time around OOM killing on 172.23.96.251 :
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.121.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.123.27.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.119.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.98.135.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.20.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.25.zip