Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45048

De-duplicate query log lines

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • query
    • Untriaged
    • 1
    • Unknown

    Description

      For the issues reported in MB-45046, on 172.23.96.251, between 2021-03-17T06:55:44 & 2021-03-17T08:21:56, which is a duration of ~1.5 hours, there are 1.5M+ occurences of lines like these in the query logs - 

      2021-03-17T08:21:56.475-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:56.486-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:56.488-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address

      This is a supportability issue as the logs get flooded by these lines and roll over quickly. It could be seen that within the millisecond, there could be as many as 12 duplicate lines logged. 

      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address 
      2021-03-17T08:21:55.463-07:00 [INFO] Pool Get returned default: dial tcp 172.23.99.21:11210: connect: cannot assign requested address

      These can be de-duplicated to avoid quick rollover of logs and flooding them with same error messages.

      Some other such lines are -
      2021-03-17T06:55:45.687-07:00 [Info] GsiClient::UpdateUsecjson: using collatejson as data format between indexer and GsiClient -> 4K+ occurences in ~2 hrs
      2021-03-17T06:55:45.610-07:00 [Info] Refreshing indexer list due to cluster changes or auto-refresh. -> 3K+ occurences

      Logs :
      Logs for the time around OOM killing on 172.23.96.251 :
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.67.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1615995907/collectinfo-2021-03-17T154508-ns_1%40172.23.99.25.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty