Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.3.0, 1.3.1, 1.3.2
    • Fix Version/s: 1.3.3
    • Component/s: library
    • Security Level: Public
    • Labels:
      None
    • Environment:

      Description

      Segmentation Fault during server rebalancing

      here is dbg backtrace:
      #0 0xb7ab6ff3 in ?? () from /usr/lib/libcouchbase.so.2
      #1 0xb7aea309 in callbacks_run (callbacks=<optimized out>) at multithread_plugin.c:460
      #2 loop_run_select (argp=141295600) at multithread_plugin.c:640
      #3 0x0805cb69 in rb_ensure (b_proc=0xb7aea010 <loop_run_select>, data1=141295600, e_proc=0xb7ae9d30 <loop_select_cleanup>, data2=138595180) at eval.c:744
      #4 0xb7ae9671 in loop_run (loop=0x85b80d0) at multithread_plugin.c:893
      #5 lcb_io_run_event_loop (iops=0x85b8070) at multithread_plugin.c:1014
      #6 0xb7ac4b25 in lcb_wait () from /usr/lib/libcouchbase.so.2
      #7 0xb7ade569 in do_loop (bucket=<optimized out>) at bucket.c:1170
      #8 do_run (args=0xbfffe85c) at bucket.c:1220
      #9 0x0805cb69 in rb_ensure (b_proc=0xb7ade480 <do_run>, data1=3221219420, e_proc=0xb7ade460 <ensure_run>, data2=3221219420) at eval.c:744
      #10 0xb7ae0b72 in cb_bucket_run (argc=0, argv=0xb7b220b4, self=138262560) at bucket.c:1307
      #11 0x0814ee55 in call_cfunc (func=0xb7ae0ae0 <cb_bucket_run>, recv=<optimized out>, len=-1, argc=0, argv=0xb7b220b4) at vm_insnhelper.c:317
      #12 0x0815e864 in vm_call_cfunc (me=0x86ac988, blockptr=0xb7ba1e0c, recv=<optimized out>, num=<optimized out>, reg_cfp=0xb7ba1df8, th=0x8269b88) at vm_insnhelper.c:404
      #13 vm_call_method (th=0x8269b88, cfp=0xb7ba1df8, num=<optimized out>, blockptr=0xb7ba1e0c, flag=0, id=9936, me=0x86ac988, recv=138262560) at vm_insnhelper.c:530
      #14 0x08153a4b in vm_exec_core (th=0x8269b88, initial=<optimized out>) at insns.def:1018
      #15 0x08158ddc in vm_exec (th=0x8269b88) at vm.c:1236
      #16 0x08159c01 in invoke_block_from_c (th=0x8269b88, block=<optimized out>, self=138264140, argc=0, argv=0x0, blockptr=0x0, cref=0x0) at vm.c:640
      #17 0x0815ee4e in vm_yield (th=<optimized out>, argv=0x0, argc=0) at vm.c:670
      #18 rb_yield_0 (argv=0x0, argc=0) at vm_eval.c:777
      #19 loop_i () at vm_eval.c:835
      #20 0x0805c831 in rb_rescue2 (b_proc=0x815ee20 <loop_i>, data1=0, r_proc=0, data2=0) at eval.c:647
      #21 0x0814f061 in rb_f_loop (self=138264140) at vm_eval.c:863
      #22 0x0815e864 in vm_call_cfunc (me=0x82b4d30, blockptr=0xb7ba1ebc, recv=<optimized out>, num=<optimized out>, reg_cfp=0xb7ba1ea8, th=0x8269b88) at vm_insnhelper.c:404
      #23 vm_call_method (th=0x8269b88, cfp=0xb7ba1ea8, num=<optimized out>, blockptr=0xb7ba1ebc, flag=8, id=2904, me=0x82b4d30, recv=138264140) at vm_insnhelper.c:530
      #24 0x08153a4b in vm_exec_core (th=0x8269b88, initial=<optimized out>) at insns.def:1018
      #25 0x08158ddc in vm_exec (th=0x8269b88) at vm.c:1236
      #26 0x08159c01 in invoke_block_from_c (th=0x8269b88, block=<optimized out>, self=136958720, argc=0, argv=0x0, blockptr=0x0, cref=0x0) at vm.c:640
      #27 0x0815ee4e in vm_yield (th=<optimized out>, argv=0x0, argc=0) at vm.c:670
      #28 rb_yield_0 (argv=0x0, argc=0) at vm_eval.c:777
      #29 loop_i () at vm_eval.c:835
      #30 0x0805c831 in rb_rescue2 (b_proc=0x815ee20 <loop_i>, data1=0, r_proc=0, data2=0) at eval.c:647
      #31 0x0814f061 in rb_f_loop (self=136958720) at vm_eval.c:863
      #32 0x0815e864 in vm_call_cfunc (me=0x82b4d30, blockptr=0xb7ba1f98, recv=<optimized out>, num=<optimized out>, reg_cfp=0xb7ba1f84, th=0x8269b88) at vm_insnhelper.c:404
      #33 vm_call_method (th=0x8269b88, cfp=0xb7ba1f84, num=<optimized out>, blockptr=0xb7ba1f98, flag=8, id=2904, me=0x82b4d30, recv=136958720) at vm_insnhelper.c:530
      #34 0x08153a4b in vm_exec_core (th=0x8269b88, initial=<optimized out>) at insns.def:1018
      #35 0x08158ddc in vm_exec (th=0x8269b88) at vm.c:1236
      #36 0x0815f757 in rb_iseq_eval_main (iseqval=138260580) at vm.c:1478
      #37 0x0805a094 in ruby_exec_internal (n=0x83db064) at eval.c:204
      #38 0x0805c419 in ruby_exec_node (n=0x83db064) at eval.c:251
      #39 ruby_run_node (n=0x83db064) at eval.c:244
      #40 0x08059b1a in main (argc=2, argv=0xbffff724) at main.c:38

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        avsej Sergey Avseyev added a comment -

        Reproducing on my machine:

        (gdb) bt
        #0 0x00007f108c6a31e5 in __GI_raise (sig=sig@entry=0x6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        #1 0x00007f108c6a6398 in __GI_abort () at abort.c:90
        #2 0x00007f108d9495e5 in rb_bug (fmt=0x7f108d981e6b "Segmentation fault") at error.c:309
        #3 0x00007f108d879732 in sigsegv (sig=0xb, info=0x7f108ee80af0, ctx=0x7f108ee809c0) at signal.c:672
        #4 <signal handler called>
        #5 0x00007f108b6f5119 in lcb_error_handler (instance=0x0, error=LCB_CLIENT_ENOMEM, errinfo=0x0) at src/error.c:62
        #6 0x00007f108b6fe642 in lcb_sockrw_v0_read (conn=0x7f108f092d78, buf=0x7f108f092430) at src/readwrite.c:34
        #7 0x00007f108b6fe741 in lcb_sockrw_v0_slurp (conn=0x7f108f092d78, buf=0x7f108f092430) at src/readwrite.c:71
        #8 0x00007f108b7020e4 in do_read_data (c=0x7f108f092ca0, allow_read=0x2) at src/server_io.c:39
        #9 0x00007f108b7022cc in lcb_server_v0_event_handler (sock=0x8, which=0x2, arg=0x7f108f092ca0) at src/server_io.c:104
        #10 0x00007f108b92e46b in callbacks_run (callbacks=0x7f108f28dfb0) at ../../../../ext/couchbase_ext/multithread_plugin.c:460
        #11 0x00007f108b92ea30 in loop_run_select (argp=0x7f108ef5da20) at ../../../../ext/couchbase_ext/multithread_plugin.c:640
        #12 0x00007f108d7bdc83 in rb_ensure (b_proc=0x7f108b92e73a <loop_run_select>, data1=0x7f108ef5da20, e_proc=0x7f108b92ea75 <loop_select_cleanup>, data2=0x7f108f227a58) at eval.c:813
        #13 0x00007f108b92f0c5 in loop_run (loop=0x7f108f28df90) at ../../../../ext/couchbase_ext/multithread_plugin.c:896
        #14 0x00007f108b92f40f in lcb_io_run_event_loop (iops=0x7f108f28ded0) at ../../../../ext/couchbase_ext/multithread_plugin.c:1017
        #15 0x00007f108b7058e1 in lcb_wait (instance=0x7f108f3e44c0) at src/wait.c:77
        #16 0x00007f108b92d45e in cb_bucket_store (cmd=LCB_SET, argc=0x2, argv=0x7f108c56d070, self=0x7f108ee364d0) at ../../../../ext/couchbase_ext/store.c:159
        #17 0x00007f108b92d57a in cb_bucket_set (argc=0x2, argv=0x7f108c56d070, self=0x7f108ee364d0) at ../../../../ext/couchbase_ext/store.c:282
        #18 0x00007f108d8e2f50 in call_cfunc_m1 (func=0x7f108b92d550 <cb_bucket_set>, recv=0x7f108ee364d0, argc=0x2, argv=0x7f108c56d070) at vm_insnhelper.c:1325
        #19 0x00007f108d8e3b05 in vm_call_cfunc_with_frame (th=0x7f108ee16590, reg_cfp=0x7f108c66ced0, ci=0x7f108f1b6428) at vm_insnhelper.c:1469
        #20 0x00007f108d8e3c12 in vm_call_cfunc (th=0x7f108ee16590, reg_cfp=0x7f108c66ced0, ci=0x7f108f1b6428) at vm_insnhelper.c:1559
        #21 0x00007f108d8e8807 in vm_exec_core (th=0x7f108ee16590, initial=0x0) at insns.def:1017
        #22 0x00007f108d8f7a83 in vm_exec (th=0x7f108ee16590) at vm.c:1201
        #23 0x00007f108d8f652c in invoke_block_from_c (th=0x7f108ee16590, block=0x7f108c66cf90, self=0x7f108ee7a4c8, argc=0x0, argv=0x0, blockptr=0x0, cref=0x0, defined_class=0x8) at vm.c:648
        #24 0x00007f108d8f665b in vm_yield (th=0x7f108ee16590, argc=0x0, argv=0x0) at vm.c:679
        #25 0x00007f108d8f2ca3 in rb_yield_0 (argc=0x0, argv=0x0) at vm_eval.c:913
        #26 0x00007f108d8f2f51 in loop_i () at vm_eval.c:971
        #27 0x00007f108d7bd769 in rb_rescue2 (b_proc=0x7f108d8f2f3e <loop_i>, data1=0x0, r_proc=0x0, data2=0x0) at eval.c:714
        #28 0x00007f108d8f2ff4 in rb_f_loop (self=0x7f108ee7a4c8) at vm_eval.c:1005
        #29 0x00007f108d8e2f7b in call_cfunc_0 (func=0x7f108d8f2f7d <rb_f_loop>, recv=0x7f108ee7a4c8, argc=0x0, argv=0x7f108c56d048) at vm_insnhelper.c:1331
        #30 0x00007f108d8e3b05 in vm_call_cfunc_with_frame (th=0x7f108ee16590, reg_cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1469
        #31 0x00007f108d8e3c12 in vm_call_cfunc (th=0x7f108ee16590, reg_cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1559
        #32 0x00007f108d8e46e7 in vm_call_method (th=0x7f108ee16590, cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1751
        #33 0x00007f108d8e4efe in vm_call_general (th=0x7f108ee16590, reg_cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1902
        #34 0x00007f108d8e86e8 in vm_exec_core (th=0x7f108ee16590, initial=0x0) at insns.def:1002
        #35 0x00007f108d8f7a83 in vm_exec (th=0x7f108ee16590) at vm.c:1201
        #36 0x00007f108d8f88c9 in rb_iseq_eval_main (iseqval=0x7f108efefc90) at vm.c:1449
        #37 0x00007f108d7bc76b in ruby_exec_internal (n=0x7f108efefc90) at eval.c:250
        #38 0x00007f108d7bc884 in ruby_exec_node (n=0x7f108efefc90) at eval.c:315
        #39 0x00007f108d7bc857 in ruby_run_node (n=0x7f108efefc90) at eval.c:307
        #40 0x00007f108d7bab02 in main (argc=0x2, argv=0x7fff7264ec28) at main.c:36

        Show
        avsej Sergey Avseyev added a comment - Reproducing on my machine: (gdb) bt #0 0x00007f108c6a31e5 in __GI_raise (sig=sig@entry=0x6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007f108c6a6398 in __GI_abort () at abort.c:90 #2 0x00007f108d9495e5 in rb_bug (fmt=0x7f108d981e6b "Segmentation fault") at error.c:309 #3 0x00007f108d879732 in sigsegv (sig=0xb, info=0x7f108ee80af0, ctx=0x7f108ee809c0) at signal.c:672 #4 <signal handler called> #5 0x00007f108b6f5119 in lcb_error_handler (instance=0x0, error=LCB_CLIENT_ENOMEM, errinfo=0x0) at src/error.c:62 #6 0x00007f108b6fe642 in lcb_sockrw_v0_read (conn=0x7f108f092d78, buf=0x7f108f092430) at src/readwrite.c:34 #7 0x00007f108b6fe741 in lcb_sockrw_v0_slurp (conn=0x7f108f092d78, buf=0x7f108f092430) at src/readwrite.c:71 #8 0x00007f108b7020e4 in do_read_data (c=0x7f108f092ca0, allow_read=0x2) at src/server_io.c:39 #9 0x00007f108b7022cc in lcb_server_v0_event_handler (sock=0x8, which=0x2, arg=0x7f108f092ca0) at src/server_io.c:104 #10 0x00007f108b92e46b in callbacks_run (callbacks=0x7f108f28dfb0) at ../../../../ext/couchbase_ext/multithread_plugin.c:460 #11 0x00007f108b92ea30 in loop_run_select (argp=0x7f108ef5da20) at ../../../../ext/couchbase_ext/multithread_plugin.c:640 #12 0x00007f108d7bdc83 in rb_ensure (b_proc=0x7f108b92e73a <loop_run_select>, data1=0x7f108ef5da20, e_proc=0x7f108b92ea75 <loop_select_cleanup>, data2=0x7f108f227a58) at eval.c:813 #13 0x00007f108b92f0c5 in loop_run (loop=0x7f108f28df90) at ../../../../ext/couchbase_ext/multithread_plugin.c:896 #14 0x00007f108b92f40f in lcb_io_run_event_loop (iops=0x7f108f28ded0) at ../../../../ext/couchbase_ext/multithread_plugin.c:1017 #15 0x00007f108b7058e1 in lcb_wait (instance=0x7f108f3e44c0) at src/wait.c:77 #16 0x00007f108b92d45e in cb_bucket_store (cmd=LCB_SET, argc=0x2, argv=0x7f108c56d070, self=0x7f108ee364d0) at ../../../../ext/couchbase_ext/store.c:159 #17 0x00007f108b92d57a in cb_bucket_set (argc=0x2, argv=0x7f108c56d070, self=0x7f108ee364d0) at ../../../../ext/couchbase_ext/store.c:282 #18 0x00007f108d8e2f50 in call_cfunc_m1 (func=0x7f108b92d550 <cb_bucket_set>, recv=0x7f108ee364d0, argc=0x2, argv=0x7f108c56d070) at vm_insnhelper.c:1325 #19 0x00007f108d8e3b05 in vm_call_cfunc_with_frame (th=0x7f108ee16590, reg_cfp=0x7f108c66ced0, ci=0x7f108f1b6428) at vm_insnhelper.c:1469 #20 0x00007f108d8e3c12 in vm_call_cfunc (th=0x7f108ee16590, reg_cfp=0x7f108c66ced0, ci=0x7f108f1b6428) at vm_insnhelper.c:1559 #21 0x00007f108d8e8807 in vm_exec_core (th=0x7f108ee16590, initial=0x0) at insns.def:1017 #22 0x00007f108d8f7a83 in vm_exec (th=0x7f108ee16590) at vm.c:1201 #23 0x00007f108d8f652c in invoke_block_from_c (th=0x7f108ee16590, block=0x7f108c66cf90, self=0x7f108ee7a4c8, argc=0x0, argv=0x0, blockptr=0x0, cref=0x0, defined_class=0x8) at vm.c:648 #24 0x00007f108d8f665b in vm_yield (th=0x7f108ee16590, argc=0x0, argv=0x0) at vm.c:679 #25 0x00007f108d8f2ca3 in rb_yield_0 (argc=0x0, argv=0x0) at vm_eval.c:913 #26 0x00007f108d8f2f51 in loop_i () at vm_eval.c:971 #27 0x00007f108d7bd769 in rb_rescue2 (b_proc=0x7f108d8f2f3e <loop_i>, data1=0x0, r_proc=0x0, data2=0x0) at eval.c:714 #28 0x00007f108d8f2ff4 in rb_f_loop (self=0x7f108ee7a4c8) at vm_eval.c:1005 #29 0x00007f108d8e2f7b in call_cfunc_0 (func=0x7f108d8f2f7d <rb_f_loop>, recv=0x7f108ee7a4c8, argc=0x0, argv=0x7f108c56d048) at vm_insnhelper.c:1331 #30 0x00007f108d8e3b05 in vm_call_cfunc_with_frame (th=0x7f108ee16590, reg_cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1469 #31 0x00007f108d8e3c12 in vm_call_cfunc (th=0x7f108ee16590, reg_cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1559 #32 0x00007f108d8e46e7 in vm_call_method (th=0x7f108ee16590, cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1751 #33 0x00007f108d8e4efe in vm_call_general (th=0x7f108ee16590, reg_cfp=0x7f108c66cf70, ci=0x7f108f1b5738) at vm_insnhelper.c:1902 #34 0x00007f108d8e86e8 in vm_exec_core (th=0x7f108ee16590, initial=0x0) at insns.def:1002 #35 0x00007f108d8f7a83 in vm_exec (th=0x7f108ee16590) at vm.c:1201 #36 0x00007f108d8f88c9 in rb_iseq_eval_main (iseqval=0x7f108efefc90) at vm.c:1449 #37 0x00007f108d7bc76b in ruby_exec_internal (n=0x7f108efefc90) at eval.c:250 #38 0x00007f108d7bc884 in ruby_exec_node (n=0x7f108efefc90) at eval.c:315 #39 0x00007f108d7bc857 in ruby_run_node (n=0x7f108efefc90) at eval.c:307 #40 0x00007f108d7bab02 in main (argc=0x2, argv=0x7fff7264ec28) at main.c:36
        Hide
        avsej Sergey Avseyev added a comment -

        As far as issue in the default IO plugin, which shipped with the gem, you can workaround the issue by using the plugin from libcouchbase.

        The following code will choose select(2) implementation

        Couchbase.connect(:engine => :select)

        If you have installed libcouchbase2-libevent package

        Couchbase.connect(:engine => :libevent)

        I'm working on the fix now.

        Show
        avsej Sergey Avseyev added a comment - As far as issue in the default IO plugin, which shipped with the gem, you can workaround the issue by using the plugin from libcouchbase. The following code will choose select(2) implementation Couchbase.connect(:engine => :select) If you have installed libcouchbase2-libevent package Couchbase.connect(:engine => :libevent) I'm working on the fix now.
        Hide
        avsej Sergey Avseyev added a comment -

        I've found the issue. Working on fix now

        Show
        avsej Sergey Avseyev added a comment - I've found the issue. Working on fix now
        Show
        avsej Sergey Avseyev added a comment - http://review.couchbase.org/28790

          People

          • Assignee:
            avsej Sergey Avseyev
            Reporter:
            pharmazone Alexey Mirniy
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes