[system test] beam.smp ( in memcpy () from /lib64/libc.so.6 ) segmentation fault when start couchbase server

Cluster information:

• 12 centos 6.2 64bit server with 4 cores CPU
• Each server has 22 GB RAM and 150 GB disk.
• 18 GB RAM for couchbase server at each node (80% total system memmories)
• Disk format ext3 on both data and root
• Load 15 million items to both buckets
• Cluster has 2 buckets, default (8GB) and saslbucket (8GB)
• Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)

• Create cluster with 10 nodes installed couchbase server 2.0.0-1708

10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.3.162 ssd
10.3.121.22
10.3.121.24
10.3.121.25
10.3.121.23

• Data path /data
• View path /data

• Spare nodes:
  10.3.121.26
  10.3.121.27

• Do swap rebalance, add nodes 26, 27 and remove nodes 24, 25
• Rebalance. Hit bug MB-6638 and MB-6595
• Do swap rebalance again, add nodes 24, 25 and remove nodes 22, 23
• Hit bug MB-6638 again at node 24
• Kill /opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon on node 24.
• Restart couchbase server, after few minutes, erlang crashed again with stack trace below

Core file callstacks:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
[New Thread 13050]
[New Thread 13051]
[New Thread 13052]
[New Thread 13053]
[New Thread 13054]
[New Thread 13055]
[New Thread 13042]
[New Thread 13043]
[New Thread 13056]
[New Thread 13057]
[New Thread 13045]
[New Thread 13071]
[New Thread 13047]
[New Thread 13044]
[New Thread 13048]
[New Thread 13059]
[New Thread 13062]
[New Thread 13058]
[New Thread 13049]
[New Thread 13098]
[New Thread 13115]
[New Thread 13061]
[New Thread 13041]
[New Thread 13038]
[New Thread 13040]
[New Thread 13046]
[New Thread 13039]
[New Thread 13100]
[New Thread 13060]
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so
Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003e9188832e in memcpy () from /lib64/libc.so.6

Thread 29 (Thread 0x7fe695a70700 (LWP 13060)):
#0 0x00007fe691c91357 in v8::internal::VariableMap::VariableMap (this=0x7fe64418f180) at src/scopes.cc:80
#1 0x00007fe691c9298a in v8::internal::Scope::Scope (this=0x7fe64418f150, inner_scope=0x0, type=v8::internal::FUNCTION_SCOPE, scope_info=...) at src/scopes.cc:148
#2 0x00007fe691c9360c in v8::internal::Scope::DeserializeScopeChain (context=0x38211f140069, global_scope=0x7fe64418ee70) at src/scopes.cc:236
#3 0x00007fe691c2f976 in v8::internal::Parser::ParseLazy (this=0x7fe695a6ef50, info=0x7fe695a6f180, source=<value optimized out>, zone_scope=0x7fe695a6eee0) at src/parser.cc:705
#4 0x00007fe691c2fc34 in v8::internal::Parser::ParseLazy (this=0x7fe695a6ef50, info=0x7fe695a6f180) at src/parser.cc:677
#5 0x00007fe691c306cb in v8::internal::ParserApi::Parse (info=0x7fe695a6f180, parsing_flags=<value optimized out>) at src/parser.cc:5602
#6 0x00007fe691aff682 in v8::internal::Compiler::CompileLazy (info=0x7fe695a6f180) at src/compiler.cc:607
#7 0x00007fe691be9fd9 in CompileLazyHelper (function=<value optimized out>, osr_ast_id=-1, flag=v8::internal::CLEAR_EXCEPTION) at src/objects.cc:7479
#8 v8::internal::JSFunction::CompileOptimized (function=<value optimized out>, osr_ast_id=-1, flag=v8::internal::CLEAR_EXCEPTION) at src/objects.cc:7515
#9 0x00007fe691c6de85 in v8::internal::Runtime_LazyRecompile(v8::internal::Arguments, v8::internal::Isolate*) () from /opt/couchbase/lib/libv8.so
#10 0x00000d594f10424e in ?? ()
#11 0x0000000100000000 in ?? ()
#12 0x00000d594f1041c1 in ?? ()
#13 0x00007fe695a6f270 in ?? ()
#14 0x00007fe695a6f2c8 in ?? ()
#15 0x00000d594f120834 in ?? ()
#16 0x000038211f151609 in ?? ()
#17 0x0000000100000000 in ?? ()
#18 0x000038211f151609 in ?? ()
#19 0x00000d594f1207c1 in ?? ()
#20 0x0000000600000000 in ?? ()
#21 0x000038211f140069 in ?? ()
#22 0x00007fe695a6f320 in ?? ()
#23 0x00000d594f12bb4e in ?? ()
#24 0x00002d63e8f69739 in ?? ()
#25 0x00002d63e8f69789 in ?? ()
#26 0x00002d63e8f69629 in ?? ()
#27 0x000038211f1403b1 in ?? ()
#28 0x00002d63e8f695b9 in ?? ()
#29 0x00002d63e8f69629 in ?? ()
#30 0x000038211f104121 in ?? ()
#31 0x000038211f147641 in ?? ()
#32 0x000038211f140069 in ?? ()
#33 0x00007fe695a6f380 in ?? ()
#34 0x00000d594f12791f in ?? ()
#35 0x00002d63e8f69739 in ?? ()
#36 0x00002d63e8f69789 in ?? ()
#37 0x00002d63e8f69629 in ?? ()
#38 0x000001f5363097b9 in ?? ()
#39 0x000038211f104121 in ?? ()
#40 0x000038211f104121 in ?? ()
#41 0x000038211f104121 in ?? ()
#42 0x00002d63e8f69739 in ?? ()
#43 0x000038211f147411 in ?? ()
#44 0x000038211f140069 in ?? ()
#45 0x00007fe695a6f3c8 in ?? ()
#46 0x00000d594f107aae in ?? ()
#47 0x000038211f104121 in ?? ()
#48 0x000038211f104121 in ?? ()
#49 0x00002d63e8f69629 in ?? ()
#50 0x000038211f147349 in ?? ()
#51 0x0000000100000000 in ?? ()
#52 0x000038211f147411 in ?? ()
#53 0x0000000800000000 in ?? ()
#54 0x00007fe695a6f408 in ?? ()
#55 0x00000d594f120701 in ?? ()
#56 0x00002d63e8f69629 in ?? ()
#57 0x000038211f147349 in ?? ()
#58 0x000038211f147411 in ?? ()
#59 0x00000d594f120661 in ?? ()
#60 0x0000000600000000 in ?? ()
#61 0x0000000000000000 in ?? ()

Thread 28 (Thread 0x7fe6323dd700 (LWP 13100)):
#0 0x0000003e918ab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003e918df124 in usleep () from /lib64/libc.so.6
#2 0x00007fe691cb3848 in v8::internal::ContextSwitcher::Run (this=0x7fe6441373f0) at src/v8threads.cc:457
#3 0x00007fe691c33580 in v8::internal::ThreadEntry (arg=0x7fe6441373f0) at src/platform-linux.cc:745
#4 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7fe6985ed700 (LWP 13039)):
#0 0x0000003e9200e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc961020) at pthread/ethread.c:106
#3 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7fe69664f700 (LWP 13046)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48228) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48228) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7fe697bec700 (LWP 13040)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc961130) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7fe698919700 (LWP 13038)):
#0 0x0000003e918de8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29

Thread 23 (Thread 0x7fe6978d9700 (LWP 13041)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc961130) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7fe69506f700 (LWP 13061)):
#0 process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:1067
#1 0x000000000048d496 in sched_thread_func (vesdp=0x7fe697b77140) at beam/erl_process.c:3782
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc961100) at pthread/ethread.c:106
#3 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7fe632dde700 (LWP 13115)):
#0 0x0000003e918ab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003e918df124 in usleep () from /lib64/libc.so.6
#2 0x00007fe691cb3848 in v8::internal::ContextSwitcher::Run (this=0x7fe644149780) at src/v8threads.cc:457
#3 0x00007fe691c33580 in v8::internal::ThreadEntry (arg=0x7fe644149780) at src/platform-linux.cc:745
#4 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7fe693831700 (LWP 13098)):
#0 0x0000003e9200d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007fe691c3318d in v8::internal::LinuxSemaphore::Wait (this=0x7fe68c109b20) at src/platform-linux.cc:869
#2 0x00007fe691c4a45c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007fe691c34136 in v8::internal::SignalSender::Run (this=0x7fe6480375a0) at src/platform-linux.cc:1126
#4 0x00007fe691c33580 in v8::internal::ThreadEntry (arg=0x7fe6480375a0) at src/platform-linux.cc:745
#5 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7fe6965e9700 (LWP 13049)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48390) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48390) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7fe6964b7700 (LWP 13058)):
#0 0x0000003e9200f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610e0) at pthread/ethread.c:106
#3 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7fe69466e700 (LWP 13062)):
#0 0x0000003e918e62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000053de47 in check_fd_events (ps=0x7fe6986a12c8, pr=0x7fe69466d360, len=0x7fe69466db7c, utvp=0x7fe69466db60) at sys/common/erl_poll.c:1918
#2 erts_poll_wait_kp (ps=0x7fe6986a12c8, pr=0x7fe69466d360, len=0x7fe69466db7c, utvp=0x7fe69466db60) at sys/common/erl_poll.c:2053
#3 0x0000000000540efe in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161
#4 0x0000000000498551 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7fe697b85200, rq=0x7fe697b5a9c0) at beam/erl_process.c:1221
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7fe697b85200) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc961100) at pthread/ethread.c:106
#9 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fe696471700 (LWP 13059)):
#0 0x0000003e9200804d in pthread_join () from /lib64/libpthread.so.0
#1 0x00007fe691cb3887 in v8::internal::ContextSwitcher::StopPreemption () at src/v8threads.cc:445
#2 0x00007fe64365f7a3 in destroyContext (ctx=0x7fe6939e3798) at mapreduce.cc:354
#3 0x00000000005138fb in nif_resource_dtor (bin=<value optimized out>) at beam/erl_nif.c:1217
#4 0x000000000048904b in erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:303
#5 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#6 0x0000000000498a81 in delete_process (p=0x7fe639ab1ae8, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#7 continue_exit_process (p=0x7fe639ab1ae8, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#8 0x000000000051db94 in terminate_proc (c_p=0x7fe639ab1ae8, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#9 handle_error (c_p=0x7fe639ab1ae8, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#10 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#11 0x000000000048d496 in sched_thread_func (vesdp=0x7fe697b5afc0) at beam/erl_process.c:3782
#12 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc961100) at pthread/ethread.c:106
#13 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fe69660b700 (LWP 13048)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48318) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48318) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fe696693700 (LWP 13044)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48138) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48138) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48138) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fe69662d700 (LWP 13047)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e482a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e482a0) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e482a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fe643658700 (LWP 13071)):
#0 0x0000003e918ab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003e918df124 in usleep () from /lib64/libc.so.6
#2 0x00007fe64365d397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7fe695a6fae0) at pthread/ethread.c:106
#4 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fe696671700 (LWP 13045)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e481b0) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e481b0) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e481b0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fe6964d9700 (LWP 13057)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48750) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48750) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fe6964fb700 (LWP 13056)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e486d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e486d8) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e486d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fe6966b5700 (LWP 13043)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e480c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e480c0) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e480c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fe6966d7700 (LWP 13042)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48048) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48048) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fe69651d700 (LWP 13055)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48660) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48660) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48660) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fe69653f700 (LWP 13054)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e485e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e485e8) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e485e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fe696561700 (LWP 13053)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48570) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48570) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fe696583700 (LWP 13052)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e484f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e484f8) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e484f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fe6965a5700 (LWP 13051)):
#0 0x0000003e9200b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7fe696e48480) at beam/erl_threads.h:697
#3 async_get (arg=0x7fe696e48480) at beam/erl_async.c:207
#4 async_main (arg=0x7fe696e48480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#6 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003e918e5ccd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fe6965c7700 (LWP 13050)):
#0 0x0000003e9188832e in memcpy () from /lib64/libc.so.6
#1 0x000000000053c7fe in erts_gzread (file=0x24902c8, buf=<value optimized out>, len=16384) at drivers/common/gzio.c:473
#2 0x000000000054e208 in invoke_read (data=0x2489ca8) at drivers/common/efile_drv.c:966
#3 0x00000000004e8f55 in async_main (arg=0x7fe696e48408) at beam/erl_async.c:281
#4 0x0000000000584d89 in thr_wrapper (vtwd=0x7ffffc9610c0) at pthread/ethread.c:106
#5 0x0000003e920077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003e918e5ccd in clone () from /lib64/libc.so.6
--------------------------------------------------------------------------------
Module information:
/lib64/libutil.so.1:
lrwxrwxrwx. 1 root root 15 May 22 14:11 /lib64/libutil.so.1 -> libutil-2.12.so
263f938a1cf2864ba0029c81cfc79a04 /lib64/libutil.so.1
/lib64/libdl.so.2:
lrwxrwxrwx. 1 root root 13 May 22 14:11 /lib64/libdl.so.2 -> libdl-2.12.so
75ec370e23707c6d7a6120babdc48cb4 /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx. 1 root root 12 May 22 14:11 /lib64/libm.so.6 -> libm-2.12.so
3c6f235178e75bca53d5b250da2e410c /lib64/libm.so.6
/lib64/libncurses.so.5:
lrwxrwxrwx. 1 root root 17 May 21 14:28 /lib64/libncurses.so.5 -> libncurses.so.5.7
ac7c1ebe07ea75b65d9a2db7ed8da403 /lib64/libncurses.so.5
/lib64/libpthread.so.0:
lrwxrwxrwx. 1 root root 18 May 22 14:11 /lib64/libpthread.so.0 -> libpthread-2.12.so
a3f524a708cbaa49307bc6655cbcfa8f /lib64/libpthread.so.0
/lib64/librt.so.1:
lrwxrwxrwx. 1 root root 13 May 22 14:11 /lib64/librt.so.1 -> librt-2.12.so
cc725aa59b0f8156ebbfcbae9613bbd0 /lib64/librt.so.1
/lib64/libc.so.6:
lrwxrwxrwx. 1 root root 12 May 22 14:11 /lib64/libc.so.6 -> libc-2.12.so
332c852c3d0128920bbd3070744d44c5 /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx. 1 root root 10 May 22 14:11 /lib64/ld-linux-x86-64.so.2 -> ld-2.12.so
a52a7821fa66050f56643d255430199b /lib64/ld-linux-x86-64.so.2
/lib64/libtinfo.so.5:
lrwxrwxrwx. 1 root root 15 May 21 14:28 /lib64/libtinfo.so.5 -> libtinfo.so.5.7
98997b1c637a50565a880c1b66f31499 /lib64/libtinfo.so.5
/opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so:
-rwxr-xr-x. 1 bin bin 105367 Aug 30 09:12 /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
6e54a6b6dcaa9d8ad139be11e0d4b6a4 /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
/usr/lib64/libcrypto.so.6:
lrwxrwxrwx. 1 root root 19 Jul 20 14:38 /usr/lib64/libcrypto.so.6 -> libcrypto.so.0.9.8e
78dc020d4c5906ef8d38fa8772745aec /usr/lib64/libcrypto.so.6
/usr/lib64/libssl.so.6:
lrwxrwxrwx. 1 root root 16 Jul 20 14:38 /usr/lib64/libssl.so.6 -> libssl.so.0.9.8e
a02f2de91ba1e75330034bd0d52ca968 /usr/lib64/libssl.so.6
/lib64/libz.so.1:
lrwxrwxrwx. 1 root root 13 May 21 14:29 /lib64/libz.so.1 -> libz.so.1.2.3
9bf790e6b57f1b026a4a91d58512f4cd /lib64/libz.so.1
/lib64/libgssapi_krb5.so.2:
lrwxrwxrwx. 1 root root 21 May 23 13:43 /lib64/libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2
37011995e6675ddbe9887a14e995289e /lib64/libgssapi_krb5.so.2
/lib64/libkrb5.so.3:
lrwxrwxrwx. 1 root root 14 May 23 13:43 /lib64/libkrb5.so.3 -> libkrb5.so.3.3
7b750b1ee5ec3d057a20e581a9e71b99 /lib64/libkrb5.so.3
/lib64/libcom_err.so.2:
lrwxrwxrwx. 1 root root 17 May 21 14:29 /lib64/libcom_err.so.2 -> libcom_err.so.2.1
d46f3796d8ada3630d6a7b27f994e91c /lib64/libcom_err.so.2
/lib64/libk5crypto.so.3:
lrwxrwxrwx. 1 root root 18 May 23 13:43 /lib64/libk5crypto.so.3 -> libk5crypto.so.3.1
861b33de30dfb0e8ebc30baf54067765 /lib64/libk5crypto.so.3
/lib64/libkrb5support.so.0:
lrwxrwxrwx. 1 root root 21 May 23 13:43 /lib64/libkrb5support.so.0 -> libkrb5support.so.0.1
d4324277340d3dca531a40845a134573 /lib64/libkrb5support.so.0
/lib64/libkeyutils.so.1:
lrwxrwxrwx. 1 root root 18 May 21 14:29 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3
39dd55321951b25ebf352c753ebd83af /lib64/libkeyutils.so.1
/lib64/libresolv.so.2:
lrwxrwxrwx. 1 root root 17 May 22 14:11 /lib64/libresolv.so.2 -> libresolv-2.12.so
8598132d7f190b51974d3a69a063436b /lib64/libresolv.so.2
/lib64/libselinux.so.1:
-rwxr-xr-x. 1 root root 124624 Dec 7 2011 /lib64/libselinux.so.1
1e188c9af61e7269ea413805995c04bd /lib64/libselinux.so.1
/opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so:
-rwxr-xr-x. 1 bin bin 191090 Sep 14 17:26 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
65f1829bafd9bda22eed5732c41718a3 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
/opt/couchbase/lib/libv8.so:
-rwxr-xr-x. 1 bin bin 85345184 Aug 30 09:15 /opt/couchbase/lib/libv8.so
73f844b67803bb08a9de423081d12fac /opt/couchbase/lib/libv8.so
/opt/couchbase/lib/libicuuc.so.44:
lrwxrwxrwx. 1 bin bin 16 Sep 14 21:03 /opt/couchbase/lib/libicuuc.so.44 -> libicuuc.so.44.0
42f69b9a8effcfd45a1c70aea9dcb66a /opt/couchbase/lib/libicuuc.so.44
/opt/couchbase/lib/libicudata.so.44:
lrwxrwxrwx. 1 bin bin 18 Sep 14 21:03 /opt/couchbase/lib/libicudata.so.44 -> libicudata.so.44.0
797015967c9d62eb4ce64e861e45f500 /opt/couchbase/lib/libicudata.so.44
/opt/couchbase/lib/libicui18n.so.44:
lrwxrwxrwx. 1 bin bin 18 Sep 14 21:03 /opt/couchbase/lib/libicui18n.so.44 -> libicui18n.so.44.0
906d4788c04365237419e085f34cc982 /opt/couchbase/lib/libicui18n.so.44
/lib64/libcrypt.so.1:
lrwxrwxrwx. 1 root root 16 May 22 14:11 /lib64/libcrypt.so.1 -> libcrypt-2.12.so
85f870cd4e72fece995e50ecd690690c /lib64/libcrypt.so.1
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx. 1 root root 19 May 21 14:29 /usr/lib64/libstdc+.so.6 -> libstdc+.so.6.0.13
e973b6a74526686fa6c1072eee4b0c97 /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx. 1 root root 28 May 21 14:28 /lib64/libgcc_s.so.1 -> libgcc_s-4.4.6-20110824.so.1
57b9ca1cd0a39570a75357b364a252d5 /lib64/libgcc_s.so.1
/lib64/libfreebl3.so:
-rwxr-xr-x. 1 root root 386040 Dec 7 2011 /lib64/libfreebl3.so
d9a0c0afe16c0b47288e4f3ef11c8fab /lib64/libfreebl3.so
/opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so:
-rwxr-xr-x. 1 bin bin 411812 Sep 14 17:27 /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
8e1a95e6a6b69ebde5cb062b6303e7d0 /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
/opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so:
-rwxr-xr-x. 1 bin bin 655324 Sep 14 17:26 /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
790a30484c1c8f0d4f4598c2d513967a /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
/opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so:
-rwxr-xr-x. 1 bin bin 358429 Sep 14 17:26 /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
bfaff147d8eef7ce2a721b1d74c9e037 /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
/opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so:
-rwxr-xr-x. 1 bin bin 577011 Sep 14 17:26 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so
0ed1f41631963e85232351d1ae3d9852 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so