Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5524

Port server memcached on node 'ns_1@10.80.12.32' exited with status 134. Restarting. Messages: Vbucket <755> is going dead.

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.8.0
    • Fix Version/s: 2.0-beta
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Environment:
      Red Hat Enterprise Linux Server release 5.8 (Tikanga)
      Linux 2.6.18-308.1.1.el5PAE #1 SMP Fri Feb 17 16:55:17 EST 2012 i686 i686 i386 GNU/Linux
      Couchbase 1.8.0-r1

      Description

      We use to have this error with 3 nodes (Enterprise Version), with a load balancer:

      exited with status 134. Restarting. Messages: Extension support isn't implemented in this version of bucket_engine
      memcached: objectregistry.cc:51: static void ObjectRegistry::onCreateBlob(Blob*): Assertion `stats.currentSize.get() < ((size_t)1<<(sizeof(size_t)*8-1))' failed. (repeated 7 times)

      Searching on the issue tracker, we find #MB-4875 fits very well.
      We didn't realize that 2 nodes was the top possible free usage of Couchbase (we are very sorry for this, and we're are waiting the police to arrive).

      Next we setup just 2 nodes behind the load balancer. The two instances status, changed from orange to green, so, we check out the cache memory usage, and this value was very high, almost full, so we reset it.
      Now the two nodes are green, but we're seing this error, a few times:

      Port server memcached on node 'ns_1@10.80.12.32' exited with status 134. Restarting. Messages: Vbucket <755> is going dead.
      Vbucket <756> is going dead.
      Vbucket <757> is going dead.
      Vbucket <758> is going dead.
      Vbucket <759> is going dead.
      Vbucket <760> is going dead.
      Vbucket <761> is going dead.
      Vbucket <762> is going dead.
      Vbucket <763> is going dead.
      Vbucket <764> is going dead.
      Vbucket <765> is going dead.
      Vbucket <766> is going dead.
      Vbucket <767> is going dead.
      Vbucket <768> is going dead.
      Vbucket <769> is going dead.
      Vbucket <770> is going dead.
      Vbucket <771> is going dead.
      Vbucket <772> is going dead.
      Vbucket <773> is going dead.
      Vbucket <774> is going dead.
      Vbucket <775> is going dead.
      Vbucket <776> is going dead.
      Vbucket <777> is going dead.
      Vbucket <778> is going dead.
      Vbucket <779> is going dead.
      Vbucket <780> is going dead.
      Vbucket <781> is going dead.
      Vbucket <782> is going dead.
      Vbucket <783> is going dead.
      Vbucket <784> is going dead.
      Vbucket <785> is going dead.
      Vbucket <786> is going dead.
      Vbucket <787> is going dead.
      Vbucket <788> is going dead.
      Vbucket <789> is going dead.
      Vbucket <790> is going dead.
      Vbucket <791> is going dead.
      Vbucket <792> is going dead.
      Vbucket <793> is going dead.
      Vbucket <794> is going dead.
      Vbucket <795> is going dead.
      Vbucket <796> is going dead.
      Vbucket <797> is going dead.
      Vbucket <798> is going dead.
      Vbucket <799> is going dead.
      Vbucket <800> is going dead.
      Vbucket <801> is going dead.
      Vbucket <802> is going dead.
      Vbucket <803> is going dead.
      Vbucket <804> is going dead.
      Vbucket <805> is going dead.
      Vbucket <806> is going dead.
      Vbucket <807> is going dead.
      Vbucket <808> is going dead.
      Vbucket <809> is going dead.
      Vbucket <810> is going dead.
      Vbucket <811> is going dead.
      Vbucket <812> is going dead.
      Vbucket <813> is going dead.
      Vbucket <814> is going dead.
      Vbucket <815> is going dead.
      Vbucket <816> is going dead.
      Vbucket <817> is going dead.
      Vbucket <818> is going dead.
      Vbucket <819> is going dead.
      Vbucket <820> is going dead.
      Vbucket <821> is going dead.
      Vbucket <822> is going dead.
      Vbucket <823> is going dead.
      Vbucket <824> is going dead.
      Vbucket <825> is going dead.
      Vbucket <826> is going dead.
      Vbucket <827> is going dead.
      Vbucket <828> is going dead.
      Vbucket <829> is going dead.
      Vbucket <830> is going dead.
      Vbucket <831> is going dead.
      Vbucket <832> is going dead.
      Vbucket <833> is going dead.
      Vbucket <834> is going dead.
      Vbucket <835> is going dead.
      Vbucket <836> is going dead.
      Vbucket <837> is going dead.
      Vbucket <838> is going dead.
      Vbucket <839> is going dead.
      Vbucket <840> is going dead.
      Vbucket <841> is going dead.
      Vbucket <842> is going dead.
      Vbucket <843> is going dead.
      Vbucket <844> is going dead.
      Vbucket <845> is going dead.
      Vbucket <846> is going dead.
      Vbucket <847> is going dead.
      Vbucket <848> is going dead.
      Vbucket <849> is going dead.
      Vbucket <850> is going dead.
      Vbucket <851> is going dead.
      Vbucket <852> is going dead.
      Vbucket <853> is going dead.
      memcached: objectregistry.cc:51: static void ObjectRegistry::onCreateBlob(Blob*): Assertion `stats.currentSize.get() < ((size_t)1<<(sizeof(size_t)*8-1))' failed.

      But this second error, happens every time:

      Port server moxi on node 'ns_1@10.80.12.145' exited with status 139. Restarting. Messages: 2012-06-12 15:55:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
      2012-06-12 15:55:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (14)

      i'm adding new data as we seing it.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        can you also attach diags from these two nodes
        you can run cbcollect_info command to create the zip file and upload it here

        Show
        farshid Farshid Ghods (Inactive) added a comment - can you also attach diags from these two nodes you can run cbcollect_info command to create the zip file and upload it here
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        >>o, we check out the cache memory usage, and this value was very high, almost full, so we reset it.

        are you using memcached buckets ?
        did you run flush commnad ? how much data did you have in the bucket before resetting the bucket ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - >>o, we check out the cache memory usage, and this value was very high, almost full, so we reset it. are you using memcached buckets ? did you run flush commnad ? how much data did you have in the bucket before resetting the bucket ?
        Show
        rfuentes Ricardo Fuentes added a comment - cbcollect_info: https://s3.amazonaws.com/customers.couchbase.com/LanAirlines/trace_memcache1.zip https://s3.amazonaws.com/customers.couchbase.com/LanAirlines/trace_memcache2.zip https://s3.amazonaws.com/customers.couchbase.com/LanAirlines/trace_memcache3.zip
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        looks like memcached was not shut down when you restarted couchbase server so the old process is still running and ns-server is not able spawn a new process

        i see these errors in the logs from 04:41 to 12:41

        the workaround is to run killall -9 memcached and then restart couchbase-server

        ERROR REPORT <5882.383.0> 2012-06-12 04:41:36
        ===============================================================================

          • Generic server <5882.383.0> terminating
          • Last message in was {#Port<5882.2367>,
            Unknown macro: {exit_status,71}

            }

          • When Server state == {state,#Port<5882.2367>,memcached,
            {["failed to listen on TCP port 11210: Address already in use",

        ERROR REPORT <5882.283.0> 2012-06-12 14:09:06
        ===============================================================================

          • Generic server <5882.283.0> terminating
          • Last message in was {#Port<5882.2320>,
            Unknown macro: {exit_status,71}

            }

          • When Server state == {state,#Port<5882.2320>,memcached,
            {["failed to listen on TCP port 11210: Address already in use",
        Show
        farshid Farshid Ghods (Inactive) added a comment - looks like memcached was not shut down when you restarted couchbase server so the old process is still running and ns-server is not able spawn a new process i see these errors in the logs from 04:41 to 12:41 the workaround is to run killall -9 memcached and then restart couchbase-server ERROR REPORT <5882.383.0> 2012-06-12 04:41:36 =============================================================================== Generic server <5882.383.0> terminating Last message in was {#Port<5882.2367>, Unknown macro: {exit_status,71} } When Server state == {state,#Port<5882.2367>,memcached, {["failed to listen on TCP port 11210: Address already in use", ERROR REPORT <5882.283.0> 2012-06-12 14:09:06 =============================================================================== Generic server <5882.283.0> terminating Last message in was {#Port<5882.2320>, Unknown macro: {exit_status,71} } When Server state == {state,#Port<5882.2320>,memcached, {["failed to listen on TCP port 11210: Address already in use",
        Hide
        peter peter added a comment -

        User error

        Show
        peter peter added a comment - User error

          People

          • Assignee:
            chiyoung Chiyoung Seo
            Reporter:
            rfuentes Ricardo Fuentes
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes