Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5758

memcached hung and crashed with: memcached: bucket_engine.c:831: get_engine_handle: Assertion `es' failed.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.8.1
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:

      Description

      INFO REPORT <5862.3699.0> 2012-07-02 00:46:16
      ===============================================================================

      memcached<0.3699.0>: memcached: bucket_engine.c:831: get_engine_handle: Assertion `es' failed.

      But it took another half-hour for this to get back up into Erlang?:

      INFO REPORT <5862.3699.0> 2012-07-02 01:26:58
      ===============================================================================

      ns_log: logging ns_port_server:0:Port server memcached on node 'ns_1@172.16.10.150' exited with status 134. Restarting. Messages: Suspend eq_tapq:replication_ns_1@172.16.10.37 for 5.00 secs
      Suspend eq_tapq:replication_ns_1@172.16.10.37 for 5.00 secs
      Suspend eq_tapq:replication_ns_1@172.16.10.37 for 5.00 secs
      Suspend eq_tapq:replication_ns_1@172.16.10.37 for 5.00 secs
      ...
      memcached: bucket_engine.c:831: get_engine_handle: Assertion `es' failed.

      It looks like the system started having trouble gathering stats from both the memcached process and the system-level right before the crash:

      CRASH REPORT <5862.3660.0> 2012-07-02 00:46:02
      ===============================================================================
      Crashing process
      initial_call

      {mb_master,init,['Argument__1']}

      pid <5862.3660.0>
      registered_name mb_master
      error_info
      {exit,{timeout,{gen_server,call,[ns_node_disco,nodes_wanted]}},
      [

      {gen_fsm,terminate,7}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}
      ancestors [ns_server_sup,ns_server_cluster_sup,<5862.51.0>]
      messages
      [send_heartbeat,send_heartbeat,send_heartbeat,

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        perry Perry Krug added a comment -

        It was reported that a possibly hung (or at least very long running) backup was still in process when another one was automatically triggered. Could that have caused undo load on the system leading to the Erlang processes being unable to run properly? And that leading to mb_master shutting down?

        Show
        perry Perry Krug added a comment - It was reported that a possibly hung (or at least very long running) backup was still in process when another one was automatically triggered. Could that have caused undo load on the system leading to the Erlang processes being unable to run properly? And that leading to mb_master shutting down?
        Hide
        farshid Farshid Ghods (Inactive) added a comment - - edited

        all known bucket engine crashes including this one has been fixed in 1.8.1 release.

        other known 1.7.x and 1.8.0 bucket engine crash bugs :

        Summary Assignee Reporter P Status Resolution Created Updated
        MB-3959
        bucket_engine.c:831 assert()
        Junyi Xie Steve Yen Resolved Duplicate 06/Jun/11 27/Mar/12
        Actions
        MB-3946
        bucket_engine.c:831 crash when rebalancing out a node where cluster does not have enough disk space to accomodate vbucket movements
        Junyi Xie Farshid Ghods Resolved Won't Fix 03/Jun/11 27/Mar/12
        Actions
        MB-4156
        memcached: bucket_engine.c:831: get_engine_handle: Assertion `es' failed.
        Unassigned Perry Krug Resolved Duplicate 03/Aug/11 10/Apr/12

        MB-4754
        crashed with assertion in bucket_engine - bucket_engine.c:831: get_engine_handle: Assertion `es' failed

        Show
        farshid Farshid Ghods (Inactive) added a comment - - edited all known bucket engine crashes including this one has been fixed in 1.8.1 release. other known 1.7.x and 1.8.0 bucket engine crash bugs : Summary Assignee Reporter P Status Resolution Created Updated MB-3959 bucket_engine.c:831 assert() Junyi Xie Steve Yen Resolved Duplicate 06/Jun/11 27/Mar/12 Actions MB-3946 bucket_engine.c:831 crash when rebalancing out a node where cluster does not have enough disk space to accomodate vbucket movements Junyi Xie Farshid Ghods Resolved Won't Fix 03/Jun/11 27/Mar/12 Actions MB-4156 memcached: bucket_engine.c:831: get_engine_handle: Assertion `es' failed. Unassigned Perry Krug Resolved Duplicate 03/Aug/11 10/Apr/12 MB-4754 crashed with assertion in bucket_engine - bucket_engine.c:831: get_engine_handle: Assertion `es' failed
        Hide
        perry Perry Krug added a comment -

        Great, thanks Farshid.

        Show
        perry Perry Krug added a comment - Great, thanks Farshid.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        A bunch of changes to bucket-engine and ep-engine seemingly fixed this bucket engine assertion failure

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - A bunch of changes to bucket-engine and ep-engine seemingly fixed this bucket engine assertion failure

          People

          • Assignee:
            trond Trond Norbye
            Reporter:
            perry Perry Krug
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes