Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-1218

Segfault in lcbtrace_span_add_tag_uint64

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.1
    • 3.0.2
    • None
    • None
    • 1
    • SDK17: FLE/DOC/ServerlessPF, SDK19: .N/Py Maint,FLE,Txn RFC

    Description

      When doing a client-side durable upsert in python, only when I've made the kv_tracing_threshold low enough, I get a segfault:

      (lldb) thread backtrace
      * thread #2, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
        * frame #0: 0x0000000101d9e525 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(lcbtrace_SPAN *, const char *, uint64_t *) [inlined] slist_iter_init_at(node=0x0000000100612328, iter=<unavailable>) at sllist-inl.h:70:33
          frame #1: 0x0000000101d9e511 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(lcbtrace_SPAN *, const char *, uint64_t *) [inlined] slist_iter_init(list=0x0000000100612328, iter=<unavailable>) at sllist-inl.h:78
          frame #2: 0x0000000101d9e511 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(span=<unavailable>, name="couchbase.context_info", value=0x00007ffeefbfd3f8) at span.cc:250
          frame #3: 0x0000000101cd0d44 _libcouchbase.cpython-37m-darwin.so`pycbc_propagate_context_info(span=0x00000001006170c0, dest=0x00000001006122e0) at ext.c:2566:9 [opt]
          frame #4: 0x0000000101cd10e3 _libcouchbase.cpython-37m-darwin.so`pycbc_span_report(tracer=<unavailable>, span=0x00000001006170c0) at ext.c:2761:5 [opt]
          frame #5: 0x0000000101d9d826 libcouchbase.2.dylib`::lcbtrace_span_finish(lcbtrace_SPAN *, uint64_t) [inlined] lcb::trace::Span::finish(this=0x00000001006170c0, now=<unavailable>) at span.cc:366:9
          frame #6: 0x0000000101d9d801 libcouchbase.2.dylib`::lcbtrace_span_finish(span=0x00000001006170c0, now=<unavailable>) at span.cc:71
          frame #7: 0x0000000101d7d3c0 libcouchbase.2.dylib`lcb::durability::Durset::on_poll_done(this=0x000000010181ada0) at durability.cc:250:13
          frame #8: 0x0000000101d5aacf libcouchbase.2.dylib`void invoke_callback<lcb_RESPOBSEQNO>(pkt=0x0000000100827200, instance=0x0000000101a5de90, resp=0x00007ffeefbfd5b0, cbtype=LCB_CALLBACK_OBSEQNO) at handler.cc:376:13
          frame #9: 0x0000000101d565f8 libcouchbase.2.dylib`::mcreq_dispatch_response(mc_PIPELINE *, mc_PACKET *, lcb::MemcachedResponse *, lcb_STATUS) [inlined] H_observe_seqno(pipeline=<unavailable>, request=<unavailable>, response=0x00007ffeefbfdb90, immerr=LCB_SUCCESS) at handler.cc:835:5
          frame #10: 0x0000000101d563ff libcouchbase.2.dylib`::mcreq_dispatch_response(pipeline=0x0000000100430780, req=<unavailable>, res=0x00007ffeefbfdb90, immerr=LCB_SUCCESS) at handler.cc:1174
          frame #11: 0x0000000101d8eaf9 libcouchbase.2.dylib`lcb::Server::try_read(this=0x0000000100430780, ctx=<unavailable>, ior=0x00000001005e90a8) at mcserver.cc:0
          frame #12: 0x0000000101d902ae libcouchbase.2.dylib`on_read(ctx=0x00000001005e9060, (null)=<unavailable>) at mcserver.cc:579:26
          frame #13: 0x0000000101d28cf9 libcouchbase.2.dylib`E_handler [inlined] invoke_read_cb(ctx=<unavailable>, nb=51) at ctx.c:255:5
          frame #14: 0x0000000101d28cde libcouchbase.2.dylib`E_handler(sock=<unavailable>, which=2, arg=0x00000001005e9060) at ctx.c:282
          frame #15: 0x0000000101d12122 libcouchbase.2.dylib`run_loop(io=0x0000000101a5e2f0, is_tick=0) at plugin-select.c:309:17
          frame #16: 0x0000000101d9d6dd libcouchbase.2.dylib`::lcb_wait(instance=0x0000000101a5de90, flags=<unavailable>) at wait.cc:109:5
          frame #17: 0x0000000101cdb16f _libcouchbase.cpython-37m-darwin.so`pycbc_common_vars_wait [inlined] pycbc_oputil_wait_common(self=<unavailable>, context=<unavailable>) at oputil.c:702:5 [opt]
          frame #18: 0x0000000101cdb132 _libcouchbase.cpython-37m-darwin.so`pycbc_common_vars_wait(cv=0x00007ffeefbfe120, self=0x0000000104d948c0, context=<unavailable>) at oputil.c:205 [opt]
          frame #19: 0x0000000101cde5ca _libcouchbase.cpython-37m-darwin.so`set_common(self=0x0000000104d948c0, args=<unavailable>, kwargs=0x0000000104da0be0, operation=<unavailable>, argopts=<unavailable>, context=<unavailable>) at store.c:458:15 [opt]
          frame #20: 0x0000000101cde9c3 _libcouchbase.cpython-37m-darwin.so`pycbc_Bucket_upsert(self=0x0000000104d948c0, args=<unavailable>, kwargs=0x0000000104da0be0) at store.c:487:1 [opt]
      

      the first frame - note that node->next = iter->cur:

      frame #0: 0x0000000101d9e525 libcouchbase.2.dylib`::lcbtrace_span_get_tag_uint64(lcbtrace_SPAN *, const char *, uint64_t *) [inlined] slist_iter_init_at(node=0x0000000100612328, iter=<unavailable>) at sllist-inl.h:70:33
         67  	    iter->removed = 0;
         68
         69  	    if (iter->cur) {
      -> 70  	        iter->next = iter->cur->next;
         71  	    } else {
         72  	        iter->next = NULL;
         73  	    }
      (lldb) p *node
      (sllist_node) $8 = {
        next = 0x6466322f32323066
      }
      (lldb) p *(node->next)
      error: Couldn't apply expression side effects : Couldn't dematerialize a result variable: couldn't read its memory
      (lldb)

      So - seems the node is maybe pointing to uninitialized memory? I don't believe we manipulate these spans (the pointers are opaque to us), so I'd suspect the issue is in LCB. Sergey Avseyev LMK what you think.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          david.kelly David Kelly added a comment -

          Similar to the issue in CCBC-1215, except that was strictly for the lcb_exists calls.  Looking at this now, hoping to see something similar...

          david.kelly David Kelly added a comment - Similar to the issue in CCBC-1215 , except that was strictly for the lcb_exists calls.  Looking at this now, hoping to see something similar...

          Build couchbase-server-7.0.0-3115 contains libcouchbase commit 7032b94 with commit message:
          CCBC-1218 Intermittent segfault in client durable store

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3115 contains libcouchbase commit 7032b94 with commit message: CCBC-1218 Intermittent segfault in client durable store

          People

            david.kelly David Kelly
            david.kelly David Kelly
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty