Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: library
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Python 2.7.4
      libcouchbase 2.2.0

      Description

      Implementation is based on yours:

      https://github.com/pavel-paulau/cbagent/blob/master/cbagent/collectors/libstats/pool.py

      During rebalance I got:
      python-couchbase: self->nremaining at src/callbacks.c:106. AbortAborted (core dumped)

      (gdb) bt
      #0 0x00007f2ada467f77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
      #1 0x00007f2ada46b5e8 in __GI_abort () at abort.c:90
      #2 0x00007f2ad6ec61e0 in pycbc_handle_assert (msg=msg@entry=0x7f2ad6ed3b04 "self->nremaining", file=file@entry=0x7f2ad6ed3aa0 "src/callbacks.c", line=line@entry=106)
      at src/exceptions.c:25
      #3 0x00007f2ad6ec7d38 in maybe_breakout (self=<optimized out>, self=<optimized out>) at src/callbacks.c:106
      #4 0x00007f2ad6ec8529 in observe_callback (instance=<optimized out>, cookie=<optimized out>, err=<optimized out>, resp=0x7f2ab3ffd8d0) at src/callbacks.c:569
      #5 0x00007f2ad6c9abc1 in observe_response_handler (server=0x7f2aa805efc0, command_data=<optimized out>, res=<optimized out>) at src/handler.c:600
      #6 0x00007f2ad6c9b398 in lcb_dispatch_response (c=0x7f2aa805efc0, ct=0x7f2ab3ffdd40, header=0x7f2aa804f000) at src/handler.c:1268
      #7 0x00007f2ad6ca32e5 in lcb_proto_parse_single (c=0x7f2aa805efc0, stop=1389655096344192000) at src/server_parse.c:215
      #8 0x00007f2ad6ca2def in do_read_data (allow_read=<optimized out>, c=<optimized out>) at src/server_io.c:45
      #9 lcb_server_v0_event_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f2aa805efc0) at src/server_io.c:107
      #10 0x00007f2ad6c941c5 in lcb_io_run_event_loop (iops=<optimized out>) at plugins/io/select/plugin-select.c:562
      #11 0x00007f2ad6ca592d in lcb_wait (instance=0x26fc130) at src/wait.c:77
      #12 0x00007f2ad6ecdeb3 in pycbc_oputil_wait_common (self=self@entry=0x225f050) at src/oputil.c:510
      #13 0x00007f2ad6ecdf30 in pycbc_common_vars_wait (cv=cv@entry=0x7f2ab3ffe000, self=self@entry=0x225f050) at src/oputil.c:59
      #14 0x00007f2ad6ed05c3 in observe_common (self=0x225f050, args=<optimized out>, kwargs=<optimized out>, argopts=<optimized out>) at src/observe.c:214
      #15 0x000000000053bf0b in PyEval_CallObjectWithKeywords ()
      #16 0x0000000000595e0c in ?? ()
      #17 0x000000000052e3cb in PyEval_EvalFrameEx ()
      #18 0x000000000052e672 in PyEval_EvalFrameEx ()
      #19 0x000000000052e672 in PyEval_EvalFrameEx ()
      #20 0x000000000050697a in ?? ()
      #21 0x00000000005301ae in PyEval_EvalFrameEx ()
      #22 0x000000000052e672 in PyEval_EvalFrameEx ()
      #23 0x000000000052e672 in PyEval_EvalFrameEx ()
      #24 0x000000000050697a in ?? ()
      #25 0x00000000004d53f4 in ?? ()
      #26 0x000000000053bf0b in PyEval_CallObjectWithKeywords ()
      #27 0x0000000000579a22 in _start ()

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          mnunberg Mark Nunberg added a comment -

          Can you try with the libevent plugin?

          Show
          mnunberg Mark Nunberg added a comment - Can you try with the libevent plugin?
          Hide
          pavelpaulau Pavel Paulau added a comment -

          Doesn't help:

          (gdb) bt
          #0 0x00007fb77861bf77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
          #1 0x00007fb77861f5e8 in __GI_abort () at abort.c:90
          #2 0x00007fb7750fc1e0 in pycbc_handle_assert (msg=msg@entry=0x7fb775109b04 "self->nremaining", file=file@entry=0x7fb775109aa0 "src/callbacks.c", line=line@entry=106)
          at src/exceptions.c:25
          #3 0x00007fb7750fdd38 in maybe_breakout (self=<optimized out>, self=<optimized out>) at src/callbacks.c:106
          #4 0x00007fb7750fe529 in observe_callback (instance=<optimized out>, cookie=<optimized out>, err=<optimized out>, resp=0x7fb760ff7c10) at src/callbacks.c:569
          #5 0x00007fb774ed74a1 in lcb_failout_observe_request (server=<optimized out>, command_data=0x7fb760ff7d80, packet=<optimized out>, npacket=<optimized out>,
          err=LCB_ETIMEDOUT) at src/server.c:65
          #6 0x00007fb774ed7f57 in failout_single_request (packet=<optimized out>, nkey=<optimized out>, keyptr=<optimized out>, error=<optimized out>, ct=<optimized out>,
          req=<optimized out>, server=<optimized out>) at src/server.c:240
          #7 purge_single_server (server=0x7fb7480428b0, error=<optimized out>, min_nonstale=<optimized out>) at src/server.c:351
          #8 0x00007fb774ed8420 in lcb_failout_server (server=0x3dfa, error=15884) at src/server.c:399
          #9 0x00007fb774ed85cd in server_timeout_handler (conn=<optimized out>, err=15884) at src/server_connect.c:180
          #10 0x00007fb772191f53 in event_base_loop () from /usr/lib/libevent-1.4.so.2
          #11 0x00007fb774edb92d in lcb_wait (instance=0x2db7930) at src/wait.c:77
          #12 0x00007fb775103eb3 in pycbc_oputil_wait_common (self=self@entry=0x2a780f8) at src/oputil.c:510
          #13 0x00007fb775103f30 in pycbc_common_vars_wait (cv=cv@entry=0x7fb760ff8000, self=self@entry=0x2a780f8) at src/oputil.c:59
          #14 0x00007fb7751065c3 in observe_common (self=0x2a780f8, args=<optimized out>, kwargs=<optimized out>, argopts=<optimized out>) at src/observe.c:214
          #15 0x000000000053bf0b in PyEval_CallObjectWithKeywords ()
          #16 0x0000000000595e0c in ?? ()
          #17 0x000000000052e3cb in PyEval_EvalFrameEx ()
          #18 0x000000000052e672 in PyEval_EvalFrameEx ()
          #19 0x000000000052e672 in PyEval_EvalFrameEx ()
          #20 0x000000000050697a in ?? ()
          #21 0x00000000005301ae in PyEval_EvalFrameEx ()
          #22 0x000000000052e672 in PyEval_EvalFrameEx ()
          #23 0x000000000052e672 in PyEval_EvalFrameEx ()
          #24 0x000000000050697a in ?? ()
          #25 0x00000000004d53f4 in ?? ()
          #26 0x000000000053bf0b in PyEval_CallObjectWithKeywords ()
          #27 0x0000000000579a22 in _start ()

          Show
          pavelpaulau Pavel Paulau added a comment - Doesn't help: (gdb) bt #0 0x00007fb77861bf77 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007fb77861f5e8 in __GI_abort () at abort.c:90 #2 0x00007fb7750fc1e0 in pycbc_handle_assert (msg=msg@entry=0x7fb775109b04 "self->nremaining", file=file@entry=0x7fb775109aa0 "src/callbacks.c", line=line@entry=106) at src/exceptions.c:25 #3 0x00007fb7750fdd38 in maybe_breakout (self=<optimized out>, self=<optimized out>) at src/callbacks.c:106 #4 0x00007fb7750fe529 in observe_callback (instance=<optimized out>, cookie=<optimized out>, err=<optimized out>, resp=0x7fb760ff7c10) at src/callbacks.c:569 #5 0x00007fb774ed74a1 in lcb_failout_observe_request (server=<optimized out>, command_data=0x7fb760ff7d80, packet=<optimized out>, npacket=<optimized out>, err=LCB_ETIMEDOUT) at src/server.c:65 #6 0x00007fb774ed7f57 in failout_single_request (packet=<optimized out>, nkey=<optimized out>, keyptr=<optimized out>, error=<optimized out>, ct=<optimized out>, req=<optimized out>, server=<optimized out>) at src/server.c:240 #7 purge_single_server (server=0x7fb7480428b0, error=<optimized out>, min_nonstale=<optimized out>) at src/server.c:351 #8 0x00007fb774ed8420 in lcb_failout_server (server=0x3dfa, error=15884) at src/server.c:399 #9 0x00007fb774ed85cd in server_timeout_handler (conn=<optimized out>, err=15884) at src/server_connect.c:180 #10 0x00007fb772191f53 in event_base_loop () from /usr/lib/libevent-1.4.so.2 #11 0x00007fb774edb92d in lcb_wait (instance=0x2db7930) at src/wait.c:77 #12 0x00007fb775103eb3 in pycbc_oputil_wait_common (self=self@entry=0x2a780f8) at src/oputil.c:510 #13 0x00007fb775103f30 in pycbc_common_vars_wait (cv=cv@entry=0x7fb760ff8000, self=self@entry=0x2a780f8) at src/oputil.c:59 #14 0x00007fb7751065c3 in observe_common (self=0x2a780f8, args=<optimized out>, kwargs=<optimized out>, argopts=<optimized out>) at src/observe.c:214 #15 0x000000000053bf0b in PyEval_CallObjectWithKeywords () #16 0x0000000000595e0c in ?? () #17 0x000000000052e3cb in PyEval_EvalFrameEx () #18 0x000000000052e672 in PyEval_EvalFrameEx () #19 0x000000000052e672 in PyEval_EvalFrameEx () #20 0x000000000050697a in ?? () #21 0x00000000005301ae in PyEval_EvalFrameEx () #22 0x000000000052e672 in PyEval_EvalFrameEx () #23 0x000000000052e672 in PyEval_EvalFrameEx () #24 0x000000000050697a in ?? () #25 0x00000000004d53f4 in ?? () #26 0x000000000053bf0b in PyEval_CallObjectWithKeywords () #27 0x0000000000579a22 in _start ()
          Hide
          mnunberg Mark Nunberg added a comment -

          Then it's likely a bug specific to OBSERVE then. Are you adding a node or removing a node?

          We'll be redoing the internal packet send queue soon enough..

          Show
          mnunberg Mark Nunberg added a comment - Then it's likely a bug specific to OBSERVE then. Are you adding a node or removing a node? We'll be redoing the internal packet send queue soon enough..
          Hide
          mnunberg Mark Nunberg added a comment -

          maybe try with PYCBC_ASSERT_CONTINUE (or similar) in the environment to see if there's an exception being thrown somewhere. This basically looks like LCB telling us that an observe operation as been finished twice

          Show
          mnunberg Mark Nunberg added a comment - maybe try with PYCBC_ASSERT_CONTINUE (or similar) in the environment to see if there's an exception being thrown somewhere. This basically looks like LCB telling us that an observe operation as been finished twice
          Hide
          pavelpaulau Pavel Paulau added a comment -

          It happens in both cases (adding and removing node).

          !!! python-couchbase: Assertion failure detected..
          !!! Not aborting because os.environ['PYCBC_ASSERT_CONTINUE'] was set
          !!! Depending on what went wrong, further exceptions may
          !!! still be raised, or the program may abort due to
          !!! invalid state
          !!! (debuggers should break at pycbc_handle_assert in exceptions.c)
          !!! Assertion: 'self->nremaining' at src/callbacks.c:106
          !!! python-couchbase: Assertion failure detected..
          !!! Not aborting because os.environ['PYCBC_ASSERT_CONTINUE'] was set
          !!! Depending on what went wrong, further exceptions may
          !!! still be raised, or the program may abort due to
          !!! invalid state
          !!! (debuggers should break at pycbc_handle_assert in exceptions.c)
          !!! Assertion: 'self->nremaining == 0' at src/oputil.c:67
          Remaining count != 0. Adjusting

          Show
          pavelpaulau Pavel Paulau added a comment - It happens in both cases (adding and removing node). !!! python-couchbase: Assertion failure detected.. !!! Not aborting because os.environ ['PYCBC_ASSERT_CONTINUE'] was set !!! Depending on what went wrong, further exceptions may !!! still be raised, or the program may abort due to !!! invalid state !!! (debuggers should break at pycbc_handle_assert in exceptions.c) !!! Assertion: 'self->nremaining' at src/callbacks.c:106 !!! python-couchbase: Assertion failure detected.. !!! Not aborting because os.environ ['PYCBC_ASSERT_CONTINUE'] was set !!! Depending on what went wrong, further exceptions may !!! still be raised, or the program may abort due to !!! invalid state !!! (debuggers should break at pycbc_handle_assert in exceptions.c) !!! Assertion: 'self->nremaining == 0' at src/oputil.c:67 Remaining count != 0. Adjusting
          Hide
          pavelpaulau Pavel Paulau added a comment -

          Interesting that eventually it failed with other error:

          !!! python-couchbase: Assertion failure detected..
          !!! Not aborting because os.environ['PYCBC_ASSERT_CONTINUE'] was set
          !!! Depending on what went wrong, further exceptions may
          !!! still be raised, or the program may abort due to
          !!! invalid state
          !!! (debuggers should break at pycbc_handle_assert in exceptions.c)
          !!! Assertion: 'Py_TYPE(cookie) == &pycbc_MultiResultType' at src/callbacks.c:127
          Segmentation fault (core dumped)

          #0 maybe_breakout (self=<optimized out>, self=<optimized out>) at src/callbacks.c:106
          #1 0x00007fddb67d606d in get_common_objects (cookie=<optimized out>, key=0x7fddac3022a8, nkey=43, err=err@entry=LCB_SUCCESS, conn=conn@entry=0x7fddb3659888,
          res=res@entry=0x7fddb3659890, restype=restype@entry=4, mres=mres@entry=0x7fddb3659898) at src/callbacks.c:132
          #2 0x00007fddb67d6d63 in delete_callback (instance=<optimized out>, cookie=<optimized out>, err=LCB_SUCCESS, resp=0x7fddb3659ae0) at src/callbacks.c:380
          #3 0x00007fddb65a95d0 in delete_response_handler (res=<optimized out>, command_data=<optimized out>, server=<optimized out>) at src/handler.c:495
          #4 lcb_dispatch_response (c=0x7fddac200eb8, ct=0x7fddb3659c50, header=0x7fddac12fa80) at src/handler.c:1230
          #5 0x00007fddb65b12e5 in lcb_proto_parse_single (c=0x7fddac200eb8, stop=1389665272839153000) at src/server_parse.c:215
          #6 0x00007fddb65b0def in do_read_data (allow_read=<optimized out>, c=<optimized out>) at src/server_io.c:45
          #7 lcb_server_v0_event_handler (sock=<optimized out>, which=<optimized out>, arg=0x7fddac200eb8) at src/server_io.c:107
          #8 0x00007fddb3869f53 in event_base_loop () from /usr/lib/libevent-1.4.so.2
          #9 0x00007fddb65b392d in lcb_wait (instance=0x314e2a0) at src/wait.c:77
          #10 0x00007fddb67dbeb3 in pycbc_oputil_wait_common (self=self@entry=0x2f82e18) at src/oputil.c:510
          #11 0x00007fddb67dbf30 in pycbc_common_vars_wait (cv=cv@entry=0x7fddb3659f60, self=self@entry=0x2f82e18) at src/oputil.c:59
          #12 0x00007fddb67d9563 in set_common (self=0x2f82e18, args=<optimized out>, kwargs=<optimized out>, operation=<optimized out>, argopts=1) at src/store.c:351
          #13 0x000000000053bf0b in PyEval_CallObjectWithKeywords ()
          #14 0x0000000000595e0c in ?? ()
          #15 0x000000000052e3cb in PyEval_EvalFrameEx ()
          #16 0x0000000000505b24 in PyEval_EvalCodeEx ()
          #17 0x000000000052e237 in PyEval_EvalFrameEx ()
          #18 0x000000000052e672 in PyEval_EvalFrameEx ()
          #19 0x000000000050697a in ?? ()
          #20 0x00000000005301ae in PyEval_EvalFrameEx ()
          #21 0x000000000052e672 in PyEval_EvalFrameEx ()
          #22 0x000000000052e672 in PyEval_EvalFrameEx ()
          #23 0x000000000050697a in ?? ()
          #24 0x00000000004d53f4 in ?? ()
          #25 0x000000000053bf0b in PyEval_CallObjectWithKeywords ()
          #26 0x0000000000579a22 in _start ()

          Show
          pavelpaulau Pavel Paulau added a comment - Interesting that eventually it failed with other error: !!! python-couchbase: Assertion failure detected.. !!! Not aborting because os.environ ['PYCBC_ASSERT_CONTINUE'] was set !!! Depending on what went wrong, further exceptions may !!! still be raised, or the program may abort due to !!! invalid state !!! (debuggers should break at pycbc_handle_assert in exceptions.c) !!! Assertion: 'Py_TYPE(cookie) == &pycbc_MultiResultType' at src/callbacks.c:127 Segmentation fault (core dumped) #0 maybe_breakout (self=<optimized out>, self=<optimized out>) at src/callbacks.c:106 #1 0x00007fddb67d606d in get_common_objects (cookie=<optimized out>, key=0x7fddac3022a8, nkey=43, err=err@entry=LCB_SUCCESS, conn=conn@entry=0x7fddb3659888, res=res@entry=0x7fddb3659890, restype=restype@entry=4, mres=mres@entry=0x7fddb3659898) at src/callbacks.c:132 #2 0x00007fddb67d6d63 in delete_callback (instance=<optimized out>, cookie=<optimized out>, err=LCB_SUCCESS, resp=0x7fddb3659ae0) at src/callbacks.c:380 #3 0x00007fddb65a95d0 in delete_response_handler (res=<optimized out>, command_data=<optimized out>, server=<optimized out>) at src/handler.c:495 #4 lcb_dispatch_response (c=0x7fddac200eb8, ct=0x7fddb3659c50, header=0x7fddac12fa80) at src/handler.c:1230 #5 0x00007fddb65b12e5 in lcb_proto_parse_single (c=0x7fddac200eb8, stop=1389665272839153000) at src/server_parse.c:215 #6 0x00007fddb65b0def in do_read_data (allow_read=<optimized out>, c=<optimized out>) at src/server_io.c:45 #7 lcb_server_v0_event_handler (sock=<optimized out>, which=<optimized out>, arg=0x7fddac200eb8) at src/server_io.c:107 #8 0x00007fddb3869f53 in event_base_loop () from /usr/lib/libevent-1.4.so.2 #9 0x00007fddb65b392d in lcb_wait (instance=0x314e2a0) at src/wait.c:77 #10 0x00007fddb67dbeb3 in pycbc_oputil_wait_common (self=self@entry=0x2f82e18) at src/oputil.c:510 #11 0x00007fddb67dbf30 in pycbc_common_vars_wait (cv=cv@entry=0x7fddb3659f60, self=self@entry=0x2f82e18) at src/oputil.c:59 #12 0x00007fddb67d9563 in set_common (self=0x2f82e18, args=<optimized out>, kwargs=<optimized out>, operation=<optimized out>, argopts=1) at src/store.c:351 #13 0x000000000053bf0b in PyEval_CallObjectWithKeywords () #14 0x0000000000595e0c in ?? () #15 0x000000000052e3cb in PyEval_EvalFrameEx () #16 0x0000000000505b24 in PyEval_EvalCodeEx () #17 0x000000000052e237 in PyEval_EvalFrameEx () #18 0x000000000052e672 in PyEval_EvalFrameEx () #19 0x000000000050697a in ?? () #20 0x00000000005301ae in PyEval_EvalFrameEx () #21 0x000000000052e672 in PyEval_EvalFrameEx () #22 0x000000000052e672 in PyEval_EvalFrameEx () #23 0x000000000050697a in ?? () #24 0x00000000004d53f4 in ?? () #25 0x000000000053bf0b in PyEval_CallObjectWithKeywords () #26 0x0000000000579a22 in _start ()
          Hide
          mnunberg Mark Nunberg added a comment -

          This is a libcouchbase issue CCBC-150. Fixed in master

          Show
          mnunberg Mark Nunberg added a comment - This is a libcouchbase issue CCBC-150 . Fixed in master

            People

            • Assignee:
              mnunberg Mark Nunberg
              Reporter:
              pavelpaulau Pavel Paulau
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes