Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-1432

Slow shutdown allows one to end up with 2 memcached instances

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • 1.6.0 beta4
    • 1.6.0 beta4
    • couchbase-bucket
    • None
    • Operating System: All
      Platform: All

    Description

      From MB-1422...

      This bug is with the issue of having 2 memcached processes running, from Perry...

      There are few behavioral issues to address here, perhaps better split into
      different bugs, I'm not sure.

      Using brutis (./brutis -x 10.1.3.97 -r 1:0 -s 1024 -k 1000000 -v) I generated
      about 500k keys, and about 1GB of memory (as reported by top) into memcached.

      I then ran the 'flush_all' command, waited for it to return 'OK' and
      immediatley stopped the service (/etc/init.d/northscale-server stop).

      The memcached process remains running and spikes the CPU for some time (I'm
      assuming it's writing all the data out to disk). This is the first "undesired"
      behavior from a user's perspective. When we report that the service is
      stopped, I as the user would expect that the processes are no longer running.
      I would suggest at this point that we pause before returning the 'OK' for
      stopping the service.

      The second step involves starting the service again while the original
      memcached process is still running. You get a "Failed to start...". I tried
      it again, you get a "Warning, it is already started".

      Looking at top, I now see two memcached process, the original one and a new
      one.

      The new one begins filling up with data (I'm assuming whatever has been written
      to disk but not yet flushed) and telnetting to 11211 allows a connection but
      hangs when trying to execute any command (including stats).

      This is the second "undesired" behavior. Restarting the process should not
      start up a new memcached instance, and it certainly shouldn't fill itself with
      data that was supposed to be flushed. Of utmost importance is that if the
      instance is up, it is responsive to clients. This is not the case. For this,
      I would suggest preventing the startup of an instance if there is already one
      running.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sean@northscale.com Sean Lynch (Inactive)
              steve.yen@northscale.com Steve Yen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty