Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31402

Back off logging of connections which cannot be disconnected

    XMLWordPrintable

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 6.5.0
    • 6.5.0
    • memcached

    Description

      When memcached is trying to delete a bucket it needs to disconnect all of the clients bound to that bucket. All connections which cannot be disconnected immediately is logged every time we inspect it (which happens every second). This may cause bursts of logging messages from memcached, and in worst case fill up the entire log (if something is stuck / broken). 

       

      The current logging is possibly too aggressive and could be reduced. We cannot completely mute the logging (as we want to know if things are stuck or not).

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Trond Norbye are you planning to do this in Mad Hatter? Otherwise we should change the fix version to Cheshire Cat.

          shivani.gupta Shivani Gupta added a comment - Trond Norbye are you planning to do this in Mad Hatter? Otherwise we should change the fix version to Cheshire Cat.
          trond Trond Norbye added a comment -

          The current proposal is to do the following:

          1. Log that the bucket is being deleted
          2. After waiting 30 seconds, all connections still connected to the bucket is dumped
          3. Every 5 minutes all connections we've seen change is dumped (and the total number of blocked connections is dumped)
          4. Log that the bucket deletion completed

          In the normal case you'll only see 1 and 4. If the system is slow / overloaded you might see 2, and if you hit the state of 3 you've most likely hit a bug in our code

          trond Trond Norbye added a comment - The current proposal is to do the following: Log that the bucket is being deleted After waiting 30 seconds, all connections still connected to the bucket is dumped Every 5 minutes all connections we've seen change is dumped (and the total number of blocked connections is dumped) Log that the bucket deletion completed In the normal case you'll only see 1 and 4. If the system is slow / overloaded you might see 2, and if you hit the state of 3 you've most likely hit a bug in our code

          Build couchbase-server-6.5.0-2004 contains kv_engine commit 3cf6179 with commit message:
          Refactor: Remove logging from Connection::signalIfIdle

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2004 contains kv_engine commit 3cf6179 with commit message: Refactor: Remove logging from Connection::signalIfIdle

          Build couchbase-server-6.5.0-2041 contains kv_engine commit a2c04e2 with commit message:
          MB-31402: Back off logging of stuck connections

          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-2041 contains kv_engine commit a2c04e2 with commit message: MB-31402 : Back off logging of stuck connections

          People

            trond Trond Norbye
            trond Trond Norbye
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty