Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8486

Couchbase memcached crashed - all nodes are unhealthy

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 2.2.0
    • 2.0
    • Security Level: Public
    • Linux e-000011ed 3.2.0-31-generic #50-Ubuntu SMP Fri Sep 7 16:16:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
    • Centos 64-bit
    • Impediment

    Description

      We started to send about 150k request per minute in a cluster with 12 nodes.
      The servers started to crash. I can't figure out the reason, but I suspect it is a problem with the memcached module

      Here is a link for the cbcollect: http://docs.google.com/file/d/0B2nCT2mLzqgEWmdOM05Sc3k2NlU/edit

      This is happening every minute:
      Event Module Code Server Node Time
      Service memcached exited on node 'ns_1@172.16.74.232' in 0.45s
      supervisor_cushion001 ns_1@172.16.74.232 13:38:29 - Tue Jun 18, 2013
      Port server memcached on node 'ns_1@172.16.74.232' exited with status 134. Restarting. Messages: Tue Jun 18 12:38:29.584248 GMT+4 3: Trying to connect to mccouch: "localhost:11213"
      Tue Jun 18 12:38:29.584829 GMT+4 3: Connected to mccouch: "localhost:11213"
      Tue Jun 18 12:38:29.594811 GMT+4 3: Extension support isn't implemented in this version of bucket_engine
      Tue Jun 18 12:38:29.601103 GMT+4 3: Failed to load mutation log, falling back to key dump
      memcached: src/stored-value.cc:154: mutation_type_t HashTable::insert(const Item&, bool, bool): Assertion `itm.getCas() != static_cast(-1)' failed. ns_port_server000 ns_1@172.16.74.232 13:38:29 - Tue Jun 18, 2013
      Port server memcached on node 'ns_1@172.16.88.45' exited with status 134. Restarting. Messages: Tue Jun 18 12:38:29.622465 GMT+4 3: Trying to connect to mccouch: "localhost:11213"
      Tue Jun 18 12:38:29.623028 GMT+4 3: Connected to mccouch: "localhost:11213"
      Tue Jun 18 12:38:29.642986 GMT+4 3: Extension support isn't implemented in this version of bucket_engine
      Tue Jun 18 12:38:29.653570 GMT+4 3: Failed to load mutation log, falling back to key dump
      memcached: src/stored-value.cc:154: mutation_type_t HashTable::insert(const Item&, bool, bool): Assertion `itm.getCas() != static_cast(-1)' failed. ns_port_server000 ns_1@172.16.88.45 13:38:29 - Tue Jun 18, 2013
      Service memcached exited on node 'ns_1@172.16.75.222' in 0.51s
      supervisor_cushion001 ns_1@172.16.75.222 13:38:29 - Tue Jun 18, 2013

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            thuan Thuan Nguyen
            javier.durante Javier Durante
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 40h
                40h
                Remaining:
                Remaining Estimate - 40h
                40h
                Logged:
                Time Spent - Not Specified
                Not Specified

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty