Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5269

rebalance failed with segmentation fault

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 2.0-beta
    • 2.0-beta
    • XDCR
    • Security Level: Public
    • None
    • ubuntu 10.04 64bit

    Description

      Install couchbase server 2.0.0-1197 on ubuntu 1004 64bit
      Create a cluster of 2 nodes (102 and 103)
      Create default bucket size 3GB and load 1 million items to default bucket.
      At node 105, create default bucket size 200MB
      At cluster 102, create cluster reference node 105
      At cluster 102, create continous replication from cluster 102 to node 105.
      During xdcr running, at node 105, add node 106 and rebalance.
      Increase bucket size in cluster 105 to 2GB
      Add node 107 to cluster 105 and rebalance. Rebalance failed with segmentation fault

      -------------------------------------------------------------------------------
      File information:
      -rwxr-xr-x 1 couchbase couchbase 10145047 2012-05-08 09:51 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp
      54886b116552c6e78723595f54be5f2d /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp
      rw------ 1 couchbase couchbase 293994496 2012-05-08 19:07 /tmp/core.beam.smp.4405
      07208bc6359ca0e1c3381a5063307f08 /tmp/core.beam.smp.4405
      1.4.4_542_g19901a1
      1.4.4_542_g19901a1
      version 1.4.4_542_g19901a1
      1.4.4_542_g19901a1
      1.4.4_542_g19901a1
      1.4.4_542_g19901a1
      1.4.4_542_g19901a1
      1.4.4_542_g19901a1
      1.4.4_542_g19901a1
      1.4.4_209_g7b9e75f_eject
      --------------------------------------------------------------------------------
      Core file callstacks:
      GNU gdb (GDB) 7.1-ubuntu
      Copyright (C) 2010 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law. Type "show copying"
      and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>...
      Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
      [New Thread 4434]
      [New Thread 4426]
      [New Thread 4405]
      [New Thread 4421]
      [New Thread 4548]
      [New Thread 4417]
      [New Thread 4412]
      [New Thread 4433]
      [New Thread 4416]
      [New Thread 4410]
      [New Thread 4422]
      [New Thread 4414]
      [New Thread 4418]
      [New Thread 4423]
      [New Thread 4425]
      [New Thread 4427]
      [New Thread 4419]
      [New Thread 4428]
      [New Thread 4429]
      [New Thread 4420]
      [New Thread 4435]
      [New Thread 4430]
      [New Thread 4431]
      [New Thread 4436]
      [New Thread 4432]
      [New Thread 4424]

      warning: Can't read pathname for load map: Input/output error.
      Reading symbols from /lib/libutil.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libutil.so.1
      Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib/libdl.so.2
      Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/libm.so.6
      Reading symbols from /lib/libncurses.so.5...(no debugging symbols found)...done.
      Loaded symbols for /lib/libncurses.so.5
      Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done.
      Loaded symbols for /lib/libpthread.so.0
      Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/librt.so.1
      Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done.
      Loaded symbols for /lib/libc.so.6
      Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
      Loaded symbols for /lib64/ld-linux-x86-64.so.2
      Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done.
      Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
      Reading symbols from /usr/lib64/libcrypto.so.0.9.8...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libcrypto.so.0.9.8
      Reading symbols from /usr/lib64/libssl.so.0.9.8...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib64/libssl.so.0.9.8
      Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libz.so.1
      Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_icu_driver.so...done.
      Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_icu_driver.so
      Reading symbols from /opt/couchbase/lib/libv8.so...(no debugging symbols found)...done.
      Loaded symbols for /opt/couchbase/lib/libv8.so
      Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
      Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
      Loaded symbols for /opt/couchbase/lib/libicudata.so.44
      Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
      Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
      Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libcrypt.so.1
      Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
      Loaded symbols for /usr/lib/libstdc++.so.6
      Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done.
      Loaded symbols for /lib/libgcc_s.so.1
      Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
      Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
      Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
      Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
      Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
      Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
      Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_ejson_compare.so...done.
      Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_ejson_compare.so
      Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'.
      Program terminated with signal 11, Segmentation fault.
      #0 right_rotate (root=0x191f650, blk=0x19235f0) at beam/erl_bestfit_alloc.c:252
      252 beam/erl_bestfit_alloc.c: No such file or directory.
      in beam/erl_bestfit_alloc.c

      Thread 26 (Thread 4424):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3434, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 25 (Thread 4432):
      #0 0x00007f0f29f1248d in waitpid () from /lib/libpthread.so.0
      #1 0x000000000054f2ef in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
      #2 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #3 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #4 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #5 0x0000000000000000 in ?? ()

      Thread 24 (Thread 4436):
      #0 0x00007f0f29f0f058 in pthread_cond_signal@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb99 in ethr_cond_signal (cnd=0x7f0f283e3434) at common/ethr_mutex.c:1320
      #2 0x00000000004fc8e8 in erts_cnd_signal (ix=<value optimized out>, key=<value optimized out>, async_invoke=<value optimized out>, async_data=0x193ba18,
      async_free=0x5777e0 <free_data>) at beam/erl_threads.h:716
      #3 async_add (ix=<value optimized out>, key=<value optimized out>, async_invoke=<value optimized out>, async_data=0x193ba18, async_free=0x5777e0 <free_data>)
      at beam/erl_async.c:190
      #4 driver_async (ix=<value optimized out>, key=<value optimized out>, async_invoke=<value optimized out>, async_data=0x193ba18,
      async_free=0x5777e0 <free_data>) at beam/erl_async.c:440
      #5 0x000000000057a84d in file_outputv (e=0x192b090, ev=0x7f0f262b8c50) at drivers/common/efile_drv.c:3241
      #6 0x0000000000496f1f in erts_write_to_port (caller_id=<value optimized out>, p=0x7f0f27c793f8, list=<value optimized out>) at beam/io.c:1159
      #7 0x0000000000509231 in do_port_command (A__p=0x7f0f2988ee28, A_1=903, A_2=139702735612577, A_3=<value optimized out>, flags=<value optimized out>)
      at beam/erl_bif_port.c:181
      #8 0x0000000000537c37 in process_main () at beam/beam_emu.c:2542
      #9 0x00000000004a1d1b in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3782
      #10 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #11 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #12 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #13 0x0000000000000000 in ?? ()

      Thread 23 (Thread 4431):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e377c, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 22 (Thread 4430):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3704, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 21 (Thread 4435):
      #0 copy_shallow (ptr=<value optimized out>, sz=63, hpp=0x7f0f26ab9c28, off_heap=0x7f0f24216298) at beam/copy.c:931
      #1 0x00000000004e1c2f in db_copy_object_from_ets (p=0x7f0f24216050, ptr1=0x7f0f25627860, ptr2=0x0, tb=0x7f0f24cf0e00) at beam/erl_db_util.h:301
      #2 build_term_list (p=0x7f0f24216050, ptr1=0x7f0f25627860, ptr2=0x0, tb=0x7f0f24cf0e00) at beam/erl_db_hash.c:2480
      #3 0x00000000004e2999 in db_get_hash (p=0x7f0f24216050, tbl=0x7f0f24cf0e00, key=139702631776626, ret=0x7f0f26ab9cd8) at beam/erl_db_hash.c:873
      #4 0x00000000004d211a in ets_lookup_2 (A__p=0x7f0f24216050, A_1=648203, A_2=139702631776626) at beam/erl_db.c:1543
      #5 0x0000000000537c37 in process_main () at beam/beam_emu.c:2542
      #6 0x00000000004a1d1b in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3782
      #7 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #8 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #9 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #10 0x0000000000000000 in ?? ()

      Thread 20 (Thread 4420):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3254, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 19 (Thread 4429):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e368c, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 18 (Thread 4428):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3614, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 17 (Thread 4419):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e31dc, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 16 (Thread 4427):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e359c, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 15 (Thread 4425):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e34ac, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 14 (Thread 4423):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e33bc, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 13 (Thread 4418):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3164, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 12 (Thread 4414):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x85a064, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x0000000000476e19 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
      #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
      #4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 11 (Thread 4422):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3344, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 10 (Thread 4410):
      #0 0x00007f0f29f1193d in read () from /lib/libpthread.so.0
      #1 0x000000000054e2a0 in read (unused=<value optimized out>) at /usr/include/bits/unistd.h:45
      #2 signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
      #3 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #4 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 9 (Thread 4416):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3074, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 8 (Thread 4433):
      #0 0x00007f0f29a5ed03 in epoll_wait () from /lib/libc.so.6
      #1 0x00000000005545b9 in check_fd_events (ps=0x7f0f297a62c8, pr=<value optimized out>, len=<value optimized out>, utvp=<value optimized out>)
      at sys/common/erl_poll.c:1918
      #2 erts_poll_wait_kp (ps=0x7f0f297a62c8, pr=<value optimized out>, len=<value optimized out>, utvp=<value optimized out>) at sys/common/erl_poll.c:2053
      #3 0x00000000005576a6 in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161
      #4 0x00000000004ad0a1 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f0f28ef5fc0, rq=0x7f0f28ef53c0) at beam/erl_process.c:1221
      #5 0x00000000004af249 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
      #6 0x0000000000532682 in process_main () at beam/beam_emu.c:1225
      #7 0x00000000004a1d1b in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3782
      #8 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #9 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #10 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #11 0x0000000000000000 in ?? ()

      Thread 7 (Thread 4412):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x8a0174, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x000000000047e61f in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
      #3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
      #4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 6 (Thread 4417):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e30ec, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 5 (Thread 4548):
      #0 0x00007f0f29a2239d in nanosleep () from /lib/libc.so.6
      #1 0x00007f0f29a57844 in usleep () from /lib/libc.so.6
      #2 0x00007f0f1bd2e7e7 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:510
      #3 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #4 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #5 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #6 0x0000000000000000 in ?? ()

      Thread 4 (Thread 4421):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e32cc, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 3 (Thread 4405):
      #0 0x00007f0f29a56ff3 in select () from /lib/libc.so.6
      #1 0x000000000054dcc0 in erts_sys_main_thread () at sys/unix/sys.c:3049
      #2 0x0000000000457bf1 in erl_start (argc=49, argv=<value optimized out>) at beam/erl_init.c:1441
      #3 0x0000000000439269 in main (argc=0, argv=0x0) at sys/unix/erl_main.c:29

      Thread 2 (Thread 4426):
      #0 0x00007f0f29f0e85c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #1 0x00000000005aeb39 in ethr_cond_wait (cnd=0x7f0f283e3524, mtx=0x80) at common/ethr_mutex.c:1350
      #2 0x00000000004fca1b in erts_cnd_wait (arg=<value optimized out>) at beam/erl_threads.h:697
      #3 async_get (arg=<value optimized out>) at beam/erl_async.c:207
      #4 async_main (arg=<value optimized out>) at beam/erl_async.c:274
      #5 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #6 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #7 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #8 0x0000000000000000 in ?? ()

      Thread 1 (Thread 4434):
      #0 right_rotate (root=0x191f650, blk=0x19235f0) at beam/erl_bestfit_alloc.c:252
      #1 tree_insert_fixup (root=0x191f650, blk=0x19235f0) at beam/erl_bestfit_alloc.c:350
      #2 0x000000000045267a in bf_link_free_block (allctr=<value optimized out>, block=0x19235f0, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:764
      #3 0x000000000044b55f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=<value optimized out>) at beam/erl_alloc_util.c:2898
      #4 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=<value optimized out>) at beam/erl_alloc_util.c:2949
      #5 0x000000000057d92b in remove_multi_op (desc=<value optimized out>, id_p=0x19235f0, req_p=0x1925358, caller=<value optimized out>, timeout_p=0x1923ea8,
      monitorp=0x1152) at drivers/common/inet_drv.c:1623
      #6 0x0000000000581266 in tcp_inet_multi_timeout (e=0x191f650, caller=704387) at drivers/common/inet_drv.c:8219
      #7 0x00000000005810c8 in fire_multi_timers (e=<value optimized out>) at drivers/common/inet_drv.c:10304
      #8 tcp_inet_timeout (e=<value optimized out>) at drivers/common/inet_drv.c:8173
      #9 0x00000000004b4bde in erts_port_task_execute (runq=0x7f0f28ef55c0, curr_port_pp=<value optimized out>) at beam/erl_port_task.c:850
      #10 0x00000000004af47b in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5504
      #11 0x0000000000532682 in process_main () at beam/beam_emu.c:1225
      #12 0x00000000004a1d1b in sched_thread_func (vesdp=<value optimized out>) at beam/erl_process.c:3782
      #13 0x00000000005b0b19 in thr_wrapper (vtwd=<value optimized out>) at pthread/ethread.c:106
      #14 0x00007f0f29f099ca in start_thread () from /lib/libpthread.so.0
      #15 0x00007f0f29a5e70d in clone () from /lib/libc.so.6
      #16 0x0000000000000000 in ?? ()
      --------------------------------------------------------------------------------
      Module information:
      /lib/libutil.so.1:
      lrwxrwxrwx 1 root root 17 2012-02-13 15:47 /lib/libutil.so.1 -> libutil-2.11.1.so
      1ea2bbc550c9c02e0f828cd93271d5d8 /lib/libutil.so.1
      /lib/libdl.so.2:
      lrwxrwxrwx 1 root root 15 2012-02-13 15:47 /lib/libdl.so.2 -> libdl-2.11.1.so
      b000e5ace83cb232171a8c973c443299 /lib/libdl.so.2
      /lib/libm.so.6:
      lrwxrwxrwx 1 root root 14 2012-02-13 15:47 /lib/libm.so.6 -> libm-2.11.1.so
      77385c4ce7ee521e4c38fb9cc13f534e /lib/libm.so.6
      /lib/libncurses.so.5:
      lrwxrwxrwx 1 root root 17 2012-02-13 15:47 /lib/libncurses.so.5 -> libncurses.so.5.7
      2d9fb467603049c40ab2d2520cda84eb /lib/libncurses.so.5
      /lib/libpthread.so.0:
      lrwxrwxrwx 1 root root 20 2012-02-13 15:47 /lib/libpthread.so.0 -> libpthread-2.11.1.so
      f636a70138b6fbae3d067d560725a238 /lib/libpthread.so.0
      /lib/librt.so.1:
      lrwxrwxrwx 1 root root 15 2012-02-13 15:47 /lib/librt.so.1 -> librt-2.11.1.so
      28d8634d49bbe02e93af23a7adc632e8 /lib/librt.so.1
      /lib/libc.so.6:
      lrwxrwxrwx 1 root root 14 2012-02-13 15:47 /lib/libc.so.6 -> libc-2.11.1.so
      fbb713d16c7979bf3d7bcd93ad9eb3de /lib/libc.so.6
      /lib64/ld-linux-x86-64.so.2:
      lrwxrwxrwx 1 root root 12 2012-02-13 15:47 /lib64/ld-linux-x86-64.so.2 -> ld-2.11.1.so
      834c71fd9e194a6bf9fc501f89575472 /lib64/ld-linux-x86-64.so.2
      /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so:
      rw-rr- 1 couchbase couchbase 106714 2012-05-08 09:51 /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
      0530ce825bc8dd89da61341ad180af75 /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
      /usr/lib64/libcrypto.so.0.9.8:
      lrwxrwxrwx 1 root root 23 2012-02-13 18:00 /usr/lib64/libcrypto.so.0.9.8 -> /lib/libcrypto.so.0.9.8
      3f3e0932613988faa4247c828772c40b /usr/lib64/libcrypto.so.0.9.8
      /usr/lib64/libssl.so.0.9.8:
      lrwxrwxrwx 1 root root 20 2012-02-13 18:00 /usr/lib64/libssl.so.0.9.8 -> /lib/libssl.so.0.9.8
      c7f34cd172e8350e1cbf482a4772f2da /usr/lib64/libssl.so.0.9.8
      /lib/libz.so.1:
      lrwxrwxrwx 1 root root 15 2012-02-13 15:47 /lib/libz.so.1 -> libz.so.1.2.3.3
      6abd7af4f2752f371b0ecb7cc601c3ac /lib/libz.so.1
      /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_icu_driver.so:
      rw-rr- 1 couchbase couchbase 21682 2012-05-08 09:51 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_icu_driver.so
      d3472c77cca44efe12e68e2e81d40169 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_icu_driver.so
      /opt/couchbase/lib/libv8.so:
      rw-rr- 1 couchbase couchbase 6977510 2012-05-08 09:51 /opt/couchbase/lib/libv8.so
      c6dd9cc456efdadd8e2ceebd2e66e923 /opt/couchbase/lib/libv8.so
      /opt/couchbase/lib/libicuuc.so.44:
      lrwxrwxrwx 1 couchbase couchbase 16 2012-05-08 18:14 /opt/couchbase/lib/libicuuc.so.44 -> libicuuc.so.44.0
      b9785d0e658effab885ba3b69ee0d26a /opt/couchbase/lib/libicuuc.so.44
      /opt/couchbase/lib/libicudata.so.44:
      lrwxrwxrwx 1 couchbase couchbase 18 2012-05-08 18:14 /opt/couchbase/lib/libicudata.so.44 -> libicudata.so.44.0
      93c3bc2bf90a238425388d76543168a0 /opt/couchbase/lib/libicudata.so.44
      /opt/couchbase/lib/libicui18n.so.44:
      lrwxrwxrwx 1 couchbase couchbase 18 2012-05-08 18:14 /opt/couchbase/lib/libicui18n.so.44 -> libicui18n.so.44.0
      3b804bfaad5904dc3a77510f6ba4e175 /opt/couchbase/lib/libicui18n.so.44
      /lib/libcrypt.so.1:
      lrwxrwxrwx 1 root root 18 2012-02-13 15:47 /lib/libcrypt.so.1 -> libcrypt-2.11.1.so
      84c844e26046804a5d33585d503dbe90 /lib/libcrypt.so.1
      /usr/lib/libstdc++.so.6:
      lrwxrwxrwx 1 root root 19 2012-02-13 15:48 /usr/lib/libstdc+.so.6 -> libstdc+.so.6.0.13
      145a441e95ff795ace078ba78840b91f /usr/lib/libstdc++.so.6
      /lib/libgcc_s.so.1:
      rw-rr- 1 root root 92552 2010-03-26 17:12 /lib/libgcc_s.so.1
      2fb8dc6079174e77e399d0d5b57b11e5 /lib/libgcc_s.so.1
      /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so:
      rw-rr- 1 couchbase couchbase 120606 2012-05-08 09:51 /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
      373dd44a370de21d9ec68e9a43289f21 /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
      /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so:
      rw-rr- 1 couchbase couchbase 187670 2012-05-08 09:51 /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
      6203b05e3359227e856598b67e307a7b /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
      /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so:
      rw-rr- 1 couchbase couchbase 403329 2012-05-08 09:51 /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
      95294fc700c02487eb752807561c1721 /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
      /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_ejson_compare.so:
      rw-rr- 1 couchbase couchbase 35750 2012-05-08 09:51 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_ejson_compare.so
      f3f3e811d3cc9103e635daa5a6732eb3 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-eb88cb3-git/priv/lib/couch_ejson_compare.so

      Attachments

        1. 10.1.3.102-8091-diag.txt.gz
          548 kB
        2. 10.1.3.103-8091-diag.txt.gz
          621 kB
        3. 10.1.3.106-8091-diag.txt.gz
          1.50 MB
        4. 10.1.3.107-8091-diag.txt.gz
          471 kB
        5. 105.txt.gz
          1.87 MB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            damien damien (Inactive)
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty