Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
Security Level: Public
-
client side moxi version 1.7.0
cluster version 1.7.1
moxi command line:
3 nodes
centos 5.4 64bit
Description
1) Set up 3 node cluster
2) Start up moxi
/opt/moxi/bin/moxi -v http://10.1.5.41:8091/pools/default/bucketsStreaming/default,http://10.1.5.42:8091/pools/default/bucketsStreaming/default,http://10.1.5.134:8091/pools/default/bucketsStreaming/default -Z port_listen=11211 -u root -t 16
3) Start up set/get load against cluster
4) Start up rebalance in a loop (continually add/remove node)
5) load 1000 keys to be used for multiget
6) run ascii multiget in a loop
moxi: cproxy.c:927: cproxy_reserve_downstream: Assertion `d->timeout_tv.tv_sec == 0' failed.
Thread 19 (process 24286):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f3baf0, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f3baf0, flags=0) at event.c:1558
#3 0x0000000000408da6 in main (argc=9, argv=0x7fff83c83bf8) at memcached.c:5061
Thread 18 (process 24287):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f40640, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f40640, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 17 (process 24288):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f44c30, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f44c30, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 16 (process 24289):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f49220, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f49220, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 15 (process 24290):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f4d910, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f4d910, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 14 (process 24291):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f52000, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f52000, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 13 (process 24292):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f566f0, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f566f0, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 12 (process 24293):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5b150, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f5b150, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 11 (process 24294):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f5f8d0, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f5f8d0, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 10 (process 24295):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f641c0, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f641c0, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 9 (process 24296):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f68ab0, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f68ab0, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 8 (process 24297):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f6d3a0, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f6d3a0, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 7 (process 24298):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f71c90, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f71c90, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 6 (process 24299):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f76580, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f76580, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 5 (process 24301):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f7fb60, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f7fb60, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 4 (process 24302):
#0 0x0000003e5f4d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x0000000000447ab4 in epoll_dispatch (base=0x8f84850, tv=<value optimized out>) at epoll.c:404
#2 0x0000000000441f95 in event_base_loop (base=0x8f84850, flags=0) at event.c:1558
#3 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 3 (process 24303):
#0 0x0000003e6000ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000412a50 in assoc_maintenance_thread (arg=<value optimized out>) at assoc.c:220
#2 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 2 (process 24304):
#0 0x0000003e5f4cae46 in poll () from /lib64/libc.so.6
#1 0x000000000044fc56 in Curl_socket_ready ()
#2 0x000000000046671b in Curl_do_perform ()
#3 0x000000000044993b in run_rest_conflate (arg=0x8f8b930) at rest.c:293
#4 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6
Thread 1 (process 24300):
#0 0x0000003e5f430265 in raise () from /lib64/libc.so.6
#1 0x0000003e5f431d10 in abort () from /lib64/libc.so.6
#2 0x0000003e5f4296e6 in __assert_fail () from /lib64/libc.so.6
#3 0x000000000041a5ad in cproxy_reserve_downstream (ptd=0x90bbe00) at cproxy.c:927
#4 0x000000000041a7ae in cproxy_assign_downstream (ptd=0x90bbe00) at cproxy.c:1783
#5 0x000000000040c86b in drive_machine (c=0x92f8600) at memcached.c:1988
#6 0x0000000000442321 in event_base_loop (base=0x8f7ae70, flags=0) at event.c:1308
#7 0x00000000004137e5 in worker_libevent (arg=<value optimized out>) at thread.c:272
#8 0x0000003e600064a7 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003e5f4d3c2d in clone () from /lib64/libc.so.6