Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-1288

memcached seg faults when adding/rebalancing node under light load

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • 1.6.0 beta1.1
    • 1.6.0 beta1
    • couchbase-bucket
    • None
    • Operating System: CentOS 5.x
      Platform: X64

    Description

      I have a simple test that takes a node and then adds a new node to it by using the membase cli rebalance command. When i run this test with load supplied by memslap I noticed that right about the time that we start to add the new node, memslap exits prematurely.

      In the case for which this bug is being filed, a node on 10.253.30.47 is being added to a node on 10.253.11.144

      ['python', '/opt/NorthScale/bin/cli/membase', 'rebalance', '--cluster=10.253.11.144:8080', '--server-add=10.253.30.47:8080']

      The issue appears to be that the memcached instances on the node which is receiving the new node crashes. So in this case memcached on 10.253.11.144 crashes. Fortunately, I was able to get a core dump , which I have attached to this file. The backtrace on the core dump looks as follows.

      #0 0x00002acc8d21c4b8 in pthread_join () from /lib64/libpthread.so.0
      (gdb) bt
      #0 0x00002acc8d21c4b8 in pthread_join () from /lib64/libpthread.so.0
      #1 0x00002aaaaaad3c71 in EvpDestroy (handle=<value optimized out>) at ep_engine.h:1490
      #2 0x0000000000411fe6 in init_engine ()
      #3 0x0000000000408176 in main ()

      This occurred on a centos 5.2 x86_64 machine

      I've also attached the ns log to this bug.

      The load that was being applied to the cluster was generated as follows

      memslap -s 10.253.11.144:11211 -t 3m -v 0.50 -S 10s

      Attachments

        1. logs
          1.72 MB
          Eric Lambert
        2. core.5435.gz
          60 kB
          Eric Lambert
        3. 47_rc1.log
          638 kB
          Eric Lambert
        4. 144_rc1.log
          1.01 MB
          Eric Lambert
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sean@northscale.com Sean Lynch (Inactive)
            eric@northscale.com Eric Lambert (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty