Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-8206

memcached crashing/restarting with Assertion `metadata.size == 16' failed.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 2.1.0
    • 2.1.0
    • couchbase-bucket
    • Security Level: Public
    • None

    Description

      Encountered this issue during an xdcr test on 2.0.2 build 780.

      Created 3 buckets on local/remote site.
      Started unidirection replication from default bucket on local site to remote.
      I waited about 5 minutes after xdcr pairing to start loading data. Then I checked on the test about 30 minutes into the access phase I notices nodes were constantly restarting with these messages in UI logs:

      Port server memcached on node 'babysitter_of_ns_1@127.0.0.1' exited with status 134. Restarting.

      and from babysitter.1:

      [ns_server:info,2013-05-06T16:36:18.200,babysitter_of_ns_1@127.0.0.1:<0.134.0>:ns_port_server:log:168]memcached<0.134.0>: memcached: src/c
      ouch-kvstore/couch-kvstore.cc:1331: static int CouchKVStore::recordDbDump(Db*, DocInfo*, void*): Assertion `metadata.size == 16' failed.

      [ns_server:info,2013-05-06T16:36:18.307,babysitter_of_ns_1@127.0.0.1:<0.133.0>:supervisor_cushion:handle_info:58]Cushion managed superviso
      r for memcached failed:

      {abnormal,134}

      [ns_server:debug,2013-05-06T16:36:18.308,babysitter_of_ns_1@127.0.0.1:<0.135.0>:supervisor_cushion:init:39]starting ns_port_server with de
      lay of 5000
      [error_logger:error,2013-05-06T16:36:18.307,babysitter_of_ns_1@127.0.0.1:error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generi
      c server <0.134.0> terminating

        • Last message in was {#Port<0.3039>,
          Unknown macro: {exit_status,134}

          }

        • When Server state == {state,#Port<0.3039>,memcached,
          {["memcached: src/couch-kvstore/couch-kvstore.cc:1331: static int CouchKVStore::recordDbDump(Db*, DocInfo*,
          void*): Assertion `metadata.size == 16' failed.",
          "Mon May 6 16:36:15.660505 PDT 3: (saslbucket) metadata loaded in 2819 ms",
          "Mon May 6 16:36:13.707220 PDT 3: (default) warmup completed in 804 ms",
          "Mon May 6 16:36:13.694935 PDT 3: (default) metadata loaded in 792 ms",
          "Mon May 6 16:36:13.018266 PDT 3: (default) Failed to load mutation log, falling back to key dump",
          "Mon May 6 16:36:12.942281 PDT 3: Extension support isn't implemented in this version of bucket_engine",
          "Mon May 6 16:36:12.936368 PDT 3: (saslbucket) Failed to load mutation log, falling back to key dump",
          "Mon May 6 16:36:12.886616 PDT 3: (default) Connected to mccouch: \"127.0.0.1:11213\"",
          "Mon May 6 16:36:12.885946 PDT 3: (default) Trying to connect to mccouch: \"127.0.0.1:11213\"",
          "Mon May 6 16:36:12.879580 PDT 3: Extension support isn't implemented in this version of bucket_engine",
          "Mon May 6 16:36:12.844079 PDT 3: (saslbucket1) Failed to load mutation log, falling back to key dump",
          "Mon May 6 16:36:12.834687 PDT 3: (saslbucket) Connected to mccouch: \"127.0.0.1:11213\"",
          "Mon May 6 16:36:12.834366 PDT 3: (saslbucket) Trying to connect to mccouch: \"127.0.0.1:11213\"",
          "Mon May 6 16:36:12.810127 PDT 3: Extension support isn't implemented in this version of bucket_engine",
          "Mon May 6 16:36:12.767939 PDT 3: (saslbucket1) Connected to mccouch: \"127.0.0.1:11213\"",
          "Mon May 6 16:36:12.767549 PDT 3: (saslbucket1) Trying to connect to mccouch: \"127.0.0.1:11213\"",
          empty],

      Seems the cluster never stabilizes but memcached keeps restarting.
      Logs from suspected host attached(172.23.105.45 ), remaining logs pending.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            tommie Tommie McAfee (Inactive)
            tommie Tommie McAfee (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty