Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
1.6.5.2
-
Security Level: Public
-
None
-
Ubuntu 1004 64bit
Description
Install ms 1.6.5.2r-13 on node 52.
Load 6M keys with 2K per key to node 52
Install ms 1.6.5.2r-13 on node 53, 55 and 56.
Add node 53 to node 52
Rebalance.
After rebalance's done, kill vbuckmigrator several times. Replication worked as exptected
After replication is done, add node 55 and 56 to cluster of node 52 and 53.
Rebalance
After rebalance's done, kill vbucketmigrator serveral times
Then watch -d './stats 10.2.1.52:11211 all' shows error.
http://screencast.com/t/cDxOHgozz
Check /tmp dir, got core dump
Thread 1 (Thread 13638):
#0 reserve_cookie (cookie=0x0) at memcached.c:6121
#1 0x00007f62df56d32c in TapConnection (this=0x10b505c00,
theEngine=<value optimized out>, c=<value optimized out>,
n=<value optimized out>, f=<value optimized out>) at tapconnection.cc:68
#2 0x00007f62df5742b9 in TapConnMap::newConn (this=0x246b5a0,
cookie=0x51ef8b48, name=..., flags=20, backfillAge=0, tapKeepAlive=300)
at tapconnmap.cc:162
#3 0x00007f62df54a9db in EventuallyPersistentEngine::createTapQueue (
this=0x246b000, cookie=0x51ef8b48, client=<value optimized out>, flags=20,
userdata=0x7f62e004e7fa, nuserdata=140062641940472) at ep_engine.cc:1634
#4 0x00007f62df54ac58 in EvpGetTapIterator (handle=0x246b000,
cookie=0x51ef8b48, client=<value optimized out>,
nclient=<value optimized out>, flags=20, userdata=0x815ec04d,
nuserdata=126) at ep_engine.cc:681
#5 0x00007f62e20584bc in bucket_get_tap_iterator (handle=0x7f62e225d160,
cookie=0x51ef8b48, client=0x815ec01c, nclient=49, flags=20,
userdata=0x815ec04d, nuserdata=126) at bucket_engine.c:1122
#6 0x0000000000408883 in process_bin_tap_connect (c=0x51ef8b48)
at memcached.c:2592
#7 0x0000000000408a19 in conn_setup_tap_stream (c=0x0) at memcached.c:5419
#8 0x000000000041394e in libevent_tap_process (fd=<value optimized out>,
which=<value optimized out>, arg=0x2376900) at thread.c:485
#9 0x000000000041ca89 in event_process_active_single_queue (base=0x23a8800,
flags=<value optimized out>) at event.c:1288
#10 event_process_active (base=0x23a8800, flags=<value optimized out>)
at event.c:1355
#11 event_base_loop (base=0x23a8800, flags=<value optimized out>)
at event.c:1550
#12 0x0000000000414364 in worker_libevent (arg=0x2376900) at thread.c:254
#13 0x00007f62e3f989ca in start_thread () from /lib/libpthread.so.0
#14 0x00007f62e3cf669d in clone () from /lib/libc.so.6
#15 0x0000000000000000 in ?? ()
--------------------------------------------------------------------------------
Module information:
/opt/membase/lib/libtcmalloc_minimal.so.0:
lrwxrwxrwx 1 membase membase 28 2011-03-07 16:17 /opt/membase/lib/libtcmalloc_minimal.so.0 -> libtcmalloc_minimal.so.0.0.0
f860d74660d78069e3c8719ed91f98a3 /opt/membase/lib/libtcmalloc_minimal.so.0
/lib/librt.so.1:
lrwxrwxrwx 1 root root 15 2010-12-29 17:24 /lib/librt.so.1 -> librt-2.11.1.so
1cfe1e6a4c1630a2e050303652c3341d /lib/librt.so.1
/lib/libdl.so.2:
lrwxrwxrwx 1 root root 15 2010-12-29 17:24 /lib/libdl.so.2 -> libdl-2.11.1.so
4fc2038540e79999571aa4a338ac8692 /lib/libdl.so.2
/lib/libpthread.so.0:
lrwxrwxrwx 1 root root 20 2010-12-29 17:24 /lib/libpthread.so.0 -> libpthread-2.11.1.so
deb7c9592f790b5ec2910b627df2d480 /lib/libpthread.so.0
/lib/libc.so.6:
lrwxrwxrwx 1 root root 14 2010-12-29 17:24 /lib/libc.so.6 -> libc-2.11.1.so
df0cc88c32da17856164f20b4b82c54a /lib/libc.so.6
/lib/libm.so.6:
lrwxrwxrwx 1 root root 14 2010-12-29 17:24 /lib/libm.so.6 -> libm-2.11.1.so
54f6f084ce7ce998d5ae8efaf5daea70 /lib/libm.so.6
/usr/lib/libstdc++.so.6:
lrwxrwxrwx 1 root root 19 2010-12-29 17:24 /usr/lib/libstdc+.so.6 -> libstdc+.so.6.0.13
145a441e95ff795ace078ba78840b91f /usr/lib/libstdc++.so.6
/lib/libgcc_s.so.1:
rw-rr- 1 root root 92552 2010-03-26 17:12 /lib/libgcc_s.so.1
2fb8dc6079174e77e399d0d5b57b11e5 /lib/libgcc_s.so.1
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 12 2010-12-29 17:24 /lib64/ld-linux-x86-64.so.2 -> ld-2.11.1.so
b319f05500878fd836406046864771fa /lib64/ld-linux-x86-64.so.2
./bin/memcached/stdin_term_handler.so:
rw-rr- 1 membase membase 17142 2011-03-07 14:52 ./bin/memcached/stdin_term_handler.so
7d30ad20749f9ebe46c0a6e388036f8d ./bin/memcached/stdin_term_handler.so
./bin/bucket_engine/bucket_engine.so:
rw-rr- 1 membase membase 196697 2011-03-07 14:52 ./bin/bucket_engine/bucket_engine.so
854552d20fbead0efb15c9d8238a39e3 ./bin/bucket_engine/bucket_engine.so
bin/ep_engine/ep.so:
rw-rr- 1 membase membase 11180416 2011-03-07 14:52 bin/ep_engine/ep.so
83bedc8255d883d45e039bddb0ba8b83 bin/ep_engine/ep.so
(END)
Attachments
For Gerrit Dashboard: MB-3495 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
4825,1 | MB-3495 Don't try to release "temporary" tapconnection objects | refresh | ep-engine | Status: MERGED | +2 | +1 |
4827,1 | MB-3495 Don't try to reseve "temporary" tapconnection objects | refresh | ep-engine | Status: MERGED | +2 | +1 |
4908,1 | Merge remote branch 'membase/refresh' | master | ep-engine | Status: MERGED | +2 | +1 |