Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-1218

Segfault in lcbtrace_span_add_tag_uint64

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.1
    • 3.0.2
    • None
    • None
    • 1
    • SDK17: FLE/DOC/ServerlessPF, SDK19: .N/Py Maint,FLE,Txn RFC

    Description

      When doing a client-side durable upsert in python, only when I've made the kv_tracing_threshold low enough, I get a segfault:

      (lldb) thread backtrace
      * thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
        * frame #0: 0x0000000101d9e525 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(lcbtrace_SPAN *, const char *, uint64_t *) [inlined] slist_iter_init_at(node=0x0000000100612328, iter=<unavailable>) at sllist-inl.h:70:33
          frame #1: 0x0000000101d9e511 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(lcbtrace_SPAN *, const char *, uint64_t *) [inlined] slist_iter_init(list=0x0000000100612328, iter=<unavailable>) at sllist-inl.h:78
          frame #2: 0x0000000101d9e511 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(span=<unavailable>, name="couchbase.context_info", value=0x00007ffeefbfd3f8) at span.cc:250
          frame #3: 0x0000000101cd0d44 _libcouchbase.cpython-37m-darwin.so`pycbc_propagate_context_info(span=0x00000001006170c0, dest=0x00000001006122e0) at ext.c:2566:9 [opt]
          frame #4: 0x0000000101cd10e3 _libcouchbase.cpython-37m-darwin.so`pycbc_span_report(tracer=<unavailable>, span=0x00000001006170c0) at ext.c:2761:5 [opt]
          frame #5: 0x0000000101d9d826 libcouchbase.2.dylib`::lcbtrace_span_finish(lcbtrace_SPAN *, uint64_t) [inlined] lcb::trace::Span::finish(this=0x00000001006170c0, now=<unavailable>) at span.cc:366:9
          frame #6: 0x0000000101d9d801 libcouchbase.2.dylib`::lcbtrace_span_finish(span=0x00000001006170c0, now=<unavailable>) at span.cc:71
          frame #7: 0x0000000101d7d3c0 libcouchbase.2.dylib`lcb::durability::Durset::on_poll_done(this=0x000000010181ada0) at durability.cc:250:13
          frame #8: 0x0000000101d5aacf libcouchbase.2.dylib`void invoke_callback<lcb_RESPOBSEQNO>(pkt=0x0000000100827200, instance=0x0000000101a5de90, resp=0x00007ffeefbfd5b0, cbtype=LCB_CALLBACK_OBSEQNO) at handler.cc:376:13
          frame #9: 0x0000000101d565f8 libcouchbase.2.dylib`::mcreq_dispatch_response(mc_PIPELINE *, mc_PACKET *, lcb::MemcachedResponse *, lcb_STATUS) [inlined] H_observe_seqno(pipeline=<unavailable>, request=<unavailable>, response=0x00007ffeefbfdb90, immerr=LCB_SUCCESS) at handler.cc:835:5
          frame #10: 0x0000000101d563ff libcouchbase.2.dylib`::mcreq_dispatch_response(pipeline=0x0000000100430780, req=<unavailable>, res=0x00007ffeefbfdb90, immerr=LCB_SUCCESS) at handler.cc:1174
          frame #11: 0x0000000101d8eaf9 libcouchbase.2.dylib`lcb::Server::try_read(this=0x0000000100430780, ctx=<unavailable>, ior=0x00000001005e90a8) at mcserver.cc:0
          frame #12: 0x0000000101d902ae libcouchbase.2.dylib`on_read(ctx=0x00000001005e9060, (null)=<unavailable>) at mcserver.cc:579:26
          frame #13: 0x0000000101d28cf9 libcouchbase.2.dylib`E_handler [inlined] invoke_read_cb(ctx=<unavailable>, nb=51) at ctx.c:255:5
          frame #14: 0x0000000101d28cde libcouchbase.2.dylib`E_handler(sock=<unavailable>, which=2, arg=0x00000001005e9060) at ctx.c:282
          frame #15: 0x0000000101d12122 libcouchbase.2.dylib`run_loop(io=0x0000000101a5e2f0, is_tick=0) at plugin-select.c:309:17
          frame #16: 0x0000000101d9d6dd libcouchbase.2.dylib`::lcb_wait(instance=0x0000000101a5de90, flags=<unavailable>) at wait.cc:109:5
          frame #17: 0x0000000101cdb16f _libcouchbase.cpython-37m-darwin.so`pycbc_common_vars_wait [inlined] pycbc_oputil_wait_common(self=<unavailable>, context=<unavailable>) at oputil.c:702:5 [opt]
          frame #18: 0x0000000101cdb132 _libcouchbase.cpython-37m-darwin.so`pycbc_common_vars_wait(cv=0x00007ffeefbfe120, self=0x0000000104d948c0, context=<unavailable>) at oputil.c:205 [opt]
          frame #19: 0x0000000101cde5ca _libcouchbase.cpython-37m-darwin.so`set_common(self=0x0000000104d948c0, args=<unavailable>, kwargs=0x0000000104da0be0, operation=<unavailable>, argopts=<unavailable>, context=<unavailable>) at store.c:458:15 [opt]
          frame #20: 0x0000000101cde9c3 _libcouchbase.cpython-37m-darwin.so`pycbc_Bucket_upsert(self=0x0000000104d948c0, args=<unavailable>, kwargs=0x0000000104da0be0) at store.c:487:1 [opt]
      

      the first frame - note that node->next = iter->cur:

      frame #0: 0x0000000101d9e525 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(lcbtrace_SPAN *, const char *, uint64_t *) [inlined] slist_iter_init_at(node=0x0000000100612328, iter=<unavailable>) at sllist-inl.h:70:33
         67  	    iter->removed = 0;
         68
         69  	    if (iter->cur) {
      -> 70  	        iter->next = iter->cur->next;
         71  	    } else {
         72  	        iter->next = NULL;
         73  	    }
      (lldb) p *node
      (sllist_node) $8 = {
        next = 0x6466322f32323066
      }
      (lldb) p *(node->next)
      error: Couldn't apply expression side effects : Couldn't dematerialize a result variable: couldn't read its memory
      (lldb)

      So - seems the node is maybe pointing to uninitialized memory? I don't believe we manipulate these spans (the pointers are opaque to us), so I'd suspect the issue is in LCB. Sergey Avseyev LMK what you think.

      Attachments

        For Gerrit Dashboard: CCBC-1218
        # Subject Branch Project Status CR V

        Activity

          david.kelly David Kelly added a comment -

          Similar to the issue in CCBC-1215, except that was strictly for the lcb_exists calls.  Looking at this now, hoping to see something similar...

          david.kelly David Kelly added a comment - Similar to the issue in CCBC-1215 , except that was strictly for the lcb_exists calls.  Looking at this now, hoping to see something similar...

          Build couchbase-server-7.0.0-3115 contains libcouchbase commit 7032b94 with commit message:
          CCBC-1218 Intermittent segfault in client durable store

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3115 contains libcouchbase commit 7032b94 with commit message: CCBC-1218 Intermittent segfault in client durable store

          People

            david.kelly David Kelly
            david.kelly David Kelly
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty