Details
-
Bug
-
Resolution: Fixed
-
1.6.0 beta1
-
None
-
Operating System: CentOS 5.x
Platform: X64
Description
I have a simple test that takes a node and then adds a new node to it by using the membase cli rebalance command. When i run this test with load supplied by memslap I noticed that right about the time that we start to add the new node, memslap exits prematurely.
In the case for which this bug is being filed, a node on 10.253.30.47 is being added to a node on 10.253.11.144
['python', '/opt/NorthScale/bin/cli/membase', 'rebalance', '--cluster=10.253.11.144:8080', '--server-add=10.253.30.47:8080']
The issue appears to be that the memcached instances on the node which is receiving the new node crashes. So in this case memcached on 10.253.11.144 crashes. Fortunately, I was able to get a core dump , which I have attached to this file. The backtrace on the core dump looks as follows.
#0 0x00002acc8d21c4b8 in pthread_join () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00002acc8d21c4b8 in pthread_join () from /lib64/libpthread.so.0
#1 0x00002aaaaaad3c71 in EvpDestroy (handle=<value optimized out>) at ep_engine.h:1490
#2 0x0000000000411fe6 in init_engine ()
#3 0x0000000000408176 in main ()
This occurred on a centos 5.2 x86_64 machine
I've also attached the ns log to this bug.
The load that was being applied to the cluster was generated as follows
memslap -s 10.253.11.144:11211 -t 3m -v 0.50 -S 10s