Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45999

Closed Listener are never removed from the endpoint

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • query
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      https://github.com/couchbase/query/blob/master/server/http/service_endpoint.go#L44-L45

      We have listners as slices in the endpoint

      listener []net.Listener
      listenerTLS []net.Listener

      We only append during creation during close we never remove them. This end up accumulation. non-TLS listeners might not have issue. But TLS listeners it will problem, when certificates are refreshed.

      We should not be calling again closed network ("use of closed network connection" should never occur)

      2021/04/29 03:50:11 audit: created new audit service
      		 
      _time=2021-04-29T03:50:11.738-07:00 _level=INFO _msg=cbq-engine started version=7.0.0-N1QL datastore=http://[::1]:8091 max-concurrency=4 loglevel=INFO servicers=16 plus-servicers=64 scan-cap=512 pipeline-cap=512 pipeline-batch=16 request-cap=256 request-size-cap=67108864 max-index-api=4 max-parallelism=1 n1ql-feat-ctrl=76 use-cbo=true timeout=0s txtimeout=0s 
      _time=2021-04-29T03:50:11.739-07:00 _level=INFO _msg=Starting audit worker 2 
      _time=2021-04-29T03:50:11.739-07:00 _level=INFO _msg=Starting audit worker 3 
      _time=2021-04-29T03:50:11.739-07:00 _level=INFO _msg=Starting audit worker 1 
      _time=2021-04-29T03:50:11.739-07:00 _level=INFO _msg=Starting audit worker 4 
      _time=2021-04-29T03:50:11.739-07:00 _level=INFO _msg=Starting audit settings worker 1. 
      _time=2021-04-29T03:50:11.739-07:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2021-04-29T03:50:11.742-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - [::]:18093 
      _time=2021-04-29T03:50:11.742-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - 0.0.0.0:18093 
      _time=2021-04-29T03:50:11.742-07:00 _level=ERROR _msg=Updating node-to-node encryption level: {EncryptData:false DisableNonSSLPorts:false} 
      _time=2021-04-29T03:50:11.742-07:00 _level=INFO _msg=HttpEndpoint: Listen Address - [::]:8093 
      _time=2021-04-29T03:50:11.742-07:00 _level=INFO _msg=HttpEndpoint: Listen Address - 0.0.0.0:8093 
      _time=2021-04-29T03:50:11.763-07:00 _level=INFO _msg=Unable to initialize functions cache monitor Could not access functions change counter because <nil> 
      _time=2021-04-29T03:50:12.739-07:00 _level=INFO _msg=Starting audit update stream 
      2021/04/29 03:51:01 http: TLS handshake error from [fd63:6f75:6368:2068:188b:e5ff:fe33:259e]:49399: remote error: tls: internal error
      		 
      _time=2021-04-29T03:51:04.605-07:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2021-04-29T03:51:04.606-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - <nil> 
      _time=2021-04-29T03:51:04.606-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - <nil> 
      _time=2021-04-29T03:51:04.607-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - [::]:18093 
      _time=2021-04-29T03:51:04.607-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - 0.0.0.0:18093 
      _time=2021-04-29T03:51:12.901-07:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - <nil> 
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - <nil> 
      _time=2021-04-29T03:51:12.904-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - [::]:18093 
      _time=2021-04-29T03:51:12.904-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - 0.0.0.0:18093 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - <nil> 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - <nil> 
      _time=2021-04-29T03:51:18.885-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - [::]:18093 
      _time=2021-04-29T03:51:18.886-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - 0.0.0.0:18093 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - <nil> 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - <nil> 
      _time=2021-04-29T03:51:26.570-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - [::]:18093 
      _time=2021-04-29T03:51:26.570-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - 0.0.0.0:18093

      Look above log

      Close error twice

      _time=2021-04-29T03:51:04.607-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - [::]:18093 
      _time=2021-04-29T03:51:04.607-07:00 _level=INFO _msg=HttpEndpoint: ListenTLS Address - 0.0.0.0:18093 
      _time=2021-04-29T03:51:12.901-07:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - <nil> 
      _time=2021-04-29T03:51:12.902-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - <nil>

      second refresh 4 close

      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg= Certificates have been refreshed by ns server  
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - <nil> 
      _time=2021-04-29T03:51:18.884-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - <nil>

      third refresh 6 close

       
      		 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - close tcp6 [::]:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - close tcp4 0.0.0.0:18093: use of closed network connection 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - [::]:18093, Error - <nil> 
      _time=2021-04-29T03:51:26.569-07:00 _level=INFO _msg=HttpEndpoint: close listener, Address - 0.0.0.0:18093, Error - <nil>

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-45995 [ipv6] - Issues with client cert for ports - 18093 and 18094

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              isha Isha Kandaswamy (Inactive)
              Sitaram.Vemulapalli Sitaram Vemulapalli
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty