XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.6
    • Component/s: None
    • Labels:

      Attachments

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          Ellis.Breen Ellis Breen added a comment -

          Unfortunately got a crash today in the tracing code, perhaps because the code is expecting an input span?

          (lldb) run /Users/ellis_breen/root/virtualenvs/3.6/default/bin/nosetests couchbase.tests.cases.misc_t
          Process 71193 launched: '/Users/ellis_breen/root/virtualenvs/3.6/default/bin/python' (x86_64)
          /Users/ellis_breen/root/workspaces/couchbase/couchbase-python-client-2.3.1/couchbase/tests/base.py:301: UserWarning: Using fallback (couchbase module) defaults
          warnings.warn('Using fallback (couchbase module) defaults')
          ..../Users/ellis_breen/root/workspaces/couchbase/couchbase-python-client-2.3.1/couchbase/tests/base.py:91: DeprecationWarning: timeout keyword argument is deprecated. Use operation_timeout (with float value) in connection string instead
          return conncls(**connargs)
          .E..Process 71193 stopped
          * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
          frame #0: 0x000000010513a857 libcouchbase.2.dylib`Json::Value::resolveReference(this=0x565f434f4442558b, key="peer.latency", cend="") at lcb-jsoncpp.cpp:3161
          3158	// @param key is not null-terminated.
          3159	Value& Value::resolveReference(char const* key, char const* cend)
          3160	{
          -> 3161	JSON_ASSERT_MESSAGE(
          3162	type_ == nullValue || type_ == objectValue,
          3163	"in Json::Value::resolveReference(key, end): requires objectValue");
          3164	if (type_ == nullValue)
          Target 0: (python) stopped.
          (lldb) bt
          * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
          * frame #0: 0x000000010513a857 libcouchbase.2.dylib`Json::Value::resolveReference(this=0x565f434f4442558b, key="peer.latency", cend="") at lcb-jsoncpp.cpp:3161
          frame #1: 0x000000010512b0ab libcouchbase.2.dylib`Json::Value::operator[](this=0x565f434f4442558b, key="peer.latency") at lcb-jsoncpp.cpp:3210
          frame #2: 0x000000010510c93d libcouchbase.2.dylib`void lcb::trace::Span::add_tag<unsigned long long>(this=0x565f434f44425553, name="peer.latency", value=0) at span.cc:201
          frame #3: 0x000000010510c8f5 libcouchbase.2.dylib`::lcbtrace_span_add_tag_uint64(span=0x565f434f44425553, name="peer.latency", value=0) at span.cc:49
          frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748
          frame #5: 0x000000010508447d libcouchbase.2.dylib`::mcreq_dispatch_response(pipeline=0x0000000104400270, req=0x0000000107001800, res=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:963
          frame #6: 0x00000001050efb9d libcouchbase.2.dylib`lcb::Server::try_read(this=0x0000000104400270, ctx=0x00000001044011a0, ior=0x00000001044011e8) at mcserver.cc:399
          frame #7: 0x00000001050f1884 libcouchbase.2.dylib`on_read(ctx=0x00000001044011a0, (null)=24) at mcserver.cc:440
          frame #8: 0x000000010501cac5 libcouchbase.2.dylib`invoke_read_cb(ctx=0x00000001044011a0, nb=24) at ctx.c:274
          frame #9: 0x000000010501c729 libcouchbase.2.dylib`E_handler(sock=10, which=2, arg=0x00000001044011a0) at ctx.c:303
          frame #10: 0x000000010572f9d7 libevent_core-2.1.6.dylib`event_process_active_single_queue + 1003
          frame #11: 0x000000010572cd39 libevent_core-2.1.6.dylib`event_base_loop + 1170
          frame #12: 0x00000001043f971e libcouchbase_libevent.dylib`lcb_io_run_event_loop(iops=0x0000000100476fc0) at plugin-libevent.c:202
          frame #13: 0x000000010510c640 libcouchbase.2.dylib`::lcb_wait3(instance=0x00000001004cba70, flags=LCB_WAIT_NOCHECK) at wait.cc:140
          frame #14: 0x0000000104752cb8 _libcouchbase.cpython-36m-darwin.so`pycbc_oputil_wait_common(self=0x000000010462e750) at oputil.c:461
          frame #15: 0x0000000104752b2a _libcouchbase.cpython-36m-darwin.so`pycbc_common_vars_wait(cv=0x00007ffeefbfb6c8, self=0x000000010462e750) at oputil.c:61
          frame #16: 0x000000010474cfa4 _libcouchbase.cpython-36m-darwin.so`set_common(self=0x000000010462e750, args=0x000000010575ff88, kwargs=0x000000010570afc0, operation=3, argopts=1) at store.c:341
          frame #17: 0x000000010474d120 _libcouchbase.cpython-36m-darwin.so`pycbc_Bucket_upsert(self=0x000000010462e750, args=0x000000010575ff88, kwargs=0x000000010570afc0) at store.c:363
          frame #18: 0x00000001000e47ea Python`_PyCFunction_FastCallDict + 463
          frame #19: 0x00000001000acacc Python`_PyObject_FastCallDict + 215
          frame #20: 0x00000001000bf7f6 Python`methoddescr_call + 114
          frame #21: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
          frame #22: 0x00000001000ace3d Python`_PyObject_FastCallKeywords + 97
          frame #23: 0x0000000100148fdf Python`call_function + 441
          frame #24: 0x0000000100141f64 Python`_PyEval_EvalFrameDefault + 5082
          frame #25: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #26: 0x0000000100149ecb Python`fast_function + 227
          frame #27: 0x0000000100148fe6 Python`call_function + 448
          frame #28: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #29: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #30: 0x0000000100148fe6 Python`call_function + 448
          frame #31: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #32: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #33: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
          frame #34: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #35: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #36: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #37: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
          frame #38: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #39: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
          frame #40: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #41: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #42: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #43: 0x00000001000f6fce Python`slot_tp_call + 50
          frame #44: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
          frame #45: 0x0000000100148fdf Python`call_function + 441
          frame #46: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #47: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #48: 0x0000000100148fe6 Python`call_function + 448
          frame #49: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #50: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #51: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #52: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #53: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #54: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
          frame #55: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #56: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
          frame #57: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #58: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #59: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #60: 0x00000001000f6fce Python`slot_tp_call + 50
          frame #61: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
          frame #62: 0x0000000100148fdf Python`call_function + 441
          frame #63: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #64: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #65: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #66: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #67: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #68: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
          frame #69: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #70: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
          frame #71: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #72: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #73: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #74: 0x00000001000f6fce Python`slot_tp_call + 50
          frame #75: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
          frame #76: 0x0000000100148fdf Python`call_function + 441
          frame #77: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #78: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #79: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #80: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #81: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #82: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
          frame #83: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #84: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
          frame #85: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #86: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #87: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #88: 0x00000001000f6fce Python`slot_tp_call + 50
          frame #89: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
          frame #90: 0x0000000100148fdf Python`call_function + 441
          frame #91: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #92: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #93: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #94: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #95: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #96: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
          frame #97: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #98: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
          frame #99: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #100: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #101: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #102: 0x00000001000f6fce Python`slot_tp_call + 50
          frame #103: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
          frame #104: 0x0000000100148fdf Python`call_function + 441
          frame #105: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #106: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #107: 0x0000000100148fe6 Python`call_function + 448
          frame #108: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #109: 0x000000010014a166 Python`_PyFunction_FastCall + 121
          frame #110: 0x0000000100148fe6 Python`call_function + 448
          frame #111: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #112: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #113: 0x0000000100140b4f Python`PyEval_EvalCodeEx + 52
          frame #114: 0x00000001000ccc1b Python`function_call + 338
          frame #115: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #116: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
          frame #117: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #118: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
          frame #119: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
          frame #120: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
          frame #121: 0x00000001000ac93a Python`PyObject_Call + 101
          frame #122: 0x00000001000f78fa Python`slot_tp_init + 57
          frame #123: 0x00000001000f4846 Python`type_call + 184
          frame #124: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
          frame #125: 0x0000000100148fdf Python`call_function + 441
          frame #126: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
          frame #127: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
          frame #128: 0x0000000100140b15 Python`PyEval_EvalCode + 43
          frame #129: 0x000000010016825a Python`run_mod + 54
          frame #130: 0x000000010016853a Python`PyRun_FileExFlags + 180
          frame #131: 0x0000000100167ac7 Python`PyRun_SimpleFileExFlags + 280
          frame #132: 0x000000010017b878 Python`Py_Main + 3268
          frame #133: 0x0000000100001e1d python`___lldb_unnamed_symbol1$$python + 227
          frame #134: 0x00007fff7e325115 libdyld.dylib`start + 1

          Show
          Ellis.Breen Ellis Breen added a comment - Unfortunately got a crash today in the tracing code, perhaps because the code is expecting an input span? (lldb) run /Users/ellis_breen/root/virtualenvs/ 3.6 / default /bin/nosetests couchbase.tests.cases.misc_t Process 71193 launched: '/Users/ellis_breen/root/virtualenvs/3.6/default/bin/python' (x86_64) /Users/ellis_breen/root/workspaces/couchbase/couchbase-python-client- 2.3 . 1 /couchbase/tests/base.py: 301 : UserWarning: Using fallback (couchbase module) defaults warnings.warn( 'Using fallback (couchbase module) defaults' ) ..../Users/ellis_breen/root/workspaces/couchbase/couchbase-python-client- 2.3 . 1 /couchbase/tests/base.py: 91 : DeprecationWarning: timeout keyword argument is deprecated. Use operation_timeout (with float value) in connection string instead return conncls(**connargs) .E..Process 71193 stopped * thread # 1 , queue = 'com.apple.main-thread' , stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) frame # 0 : 0x000000010513a857 libcouchbase. 2 .dylib`Json::Value::resolveReference( this = 0x565f434f4442558b , key= "peer.latency" , cend= "" ) at lcb-jsoncpp.cpp: 3161 3158 // @param key is not null-terminated. 3159 Value& Value::resolveReference( char const * key, char const * cend) 3160 { -> 3161 JSON_ASSERT_MESSAGE( 3162 type_ == nullValue || type_ == objectValue, 3163 "in Json::Value::resolveReference(key, end): requires objectValue" ); 3164 if (type_ == nullValue) Target 0 : (python) stopped. (lldb) bt * thread # 1 , queue = 'com.apple.main-thread' , stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) * frame # 0 : 0x000000010513a857 libcouchbase. 2 .dylib`Json::Value::resolveReference( this = 0x565f434f4442558b , key= "peer.latency" , cend= "" ) at lcb-jsoncpp.cpp: 3161 frame # 1 : 0x000000010512b0ab libcouchbase. 2 .dylib`Json::Value::operator[]( this = 0x565f434f4442558b , key= "peer.latency" ) at lcb-jsoncpp.cpp: 3210 frame # 2 : 0x000000010510c93d libcouchbase. 2 .dylib` void lcb::trace::Span::add_tag<unsigned long long >( this = 0x565f434f44425553 , name= "peer.latency" , value= 0 ) at span.cc: 201 frame # 3 : 0x000000010510c8f5 libcouchbase. 2 .dylib`::lcbtrace_span_add_tag_uint64(span= 0x565f434f44425553 , name= "peer.latency" , value= 0 ) at span.cc: 49 frame # 4 : 0x00000001050859a7 libcouchbase. 2 .dylib`H_store(pipeline= 0x0000000104400270 , request= 0x0000000107001800 , response= 0x00007ffeefbfb2d0 , immerr=LCB_SUCCESS) at handler.cc: 748 frame # 5 : 0x000000010508447d libcouchbase. 2 .dylib`::mcreq_dispatch_response(pipeline= 0x0000000104400270 , req= 0x0000000107001800 , res= 0x00007ffeefbfb2d0 , immerr=LCB_SUCCESS) at handler.cc: 963 frame # 6 : 0x00000001050efb9d libcouchbase. 2 .dylib`lcb::Server::try_read( this = 0x0000000104400270 , ctx= 0x00000001044011a0 , ior= 0x00000001044011e8 ) at mcserver.cc: 399 frame # 7 : 0x00000001050f1884 libcouchbase. 2 .dylib`on_read(ctx= 0x00000001044011a0 , ( null )= 24 ) at mcserver.cc: 440 frame # 8 : 0x000000010501cac5 libcouchbase. 2 .dylib`invoke_read_cb(ctx= 0x00000001044011a0 , nb= 24 ) at ctx.c: 274 frame # 9 : 0x000000010501c729 libcouchbase. 2 .dylib`E_handler(sock= 10 , which= 2 , arg= 0x00000001044011a0 ) at ctx.c: 303 frame # 10 : 0x000000010572f9d7 libevent_core- 2.1 . 6 .dylib`event_process_active_single_queue + 1003 frame # 11 : 0x000000010572cd39 libevent_core- 2.1 . 6 .dylib`event_base_loop + 1170 frame # 12 : 0x00000001043f971e libcouchbase_libevent.dylib`lcb_io_run_event_loop(iops= 0x0000000100476fc0 ) at plugin-libevent.c: 202 frame # 13 : 0x000000010510c640 libcouchbase. 2 .dylib`::lcb_wait3(instance= 0x00000001004cba70 , flags=LCB_WAIT_NOCHECK) at wait.cc: 140 frame # 14 : 0x0000000104752cb8 _libcouchbase.cpython-36m-darwin.so`pycbc_oputil_wait_common(self= 0x000000010462e750 ) at oputil.c: 461 frame # 15 : 0x0000000104752b2a _libcouchbase.cpython-36m-darwin.so`pycbc_common_vars_wait(cv= 0x00007ffeefbfb6c8 , self= 0x000000010462e750 ) at oputil.c: 61 frame # 16 : 0x000000010474cfa4 _libcouchbase.cpython-36m-darwin.so`set_common(self= 0x000000010462e750 , args= 0x000000010575ff88 , kwargs= 0x000000010570afc0 , operation= 3 , argopts= 1 ) at store.c: 341 frame # 17 : 0x000000010474d120 _libcouchbase.cpython-36m-darwin.so`pycbc_Bucket_upsert(self= 0x000000010462e750 , args= 0x000000010575ff88 , kwargs= 0x000000010570afc0 ) at store.c: 363 frame # 18 : 0x00000001000e47ea Python`_PyCFunction_FastCallDict + 463 frame # 19 : 0x00000001000acacc Python`_PyObject_FastCallDict + 215 frame # 20 : 0x00000001000bf7f6 Python`methoddescr_call + 114 frame # 21 : 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame # 22 : 0x00000001000ace3d Python`_PyObject_FastCallKeywords + 97 frame # 23 : 0x0000000100148fdf Python`call_function + 441 frame # 24 : 0x0000000100141f64 Python`_PyEval_EvalFrameDefault + 5082 frame # 25 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 26 : 0x0000000100149ecb Python`fast_function + 227 frame # 27 : 0x0000000100148fe6 Python`call_function + 448 frame # 28 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 29 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 30 : 0x0000000100148fe6 Python`call_function + 448 frame # 31 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 32 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 33 : 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame # 34 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 35 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 36 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 37 : 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame # 38 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 39 : 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame # 40 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 41 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 42 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 43 : 0x00000001000f6fce Python`slot_tp_call + 50 frame # 44 : 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame # 45 : 0x0000000100148fdf Python`call_function + 441 frame # 46 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 47 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 48 : 0x0000000100148fe6 Python`call_function + 448 frame # 49 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 50 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 51 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 52 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 53 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 54 : 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame # 55 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 56 : 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame # 57 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 58 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 59 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 60 : 0x00000001000f6fce Python`slot_tp_call + 50 frame # 61 : 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame # 62 : 0x0000000100148fdf Python`call_function + 441 frame # 63 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 64 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 65 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 66 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 67 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 68 : 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame # 69 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 70 : 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame # 71 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 72 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 73 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 74 : 0x00000001000f6fce Python`slot_tp_call + 50 frame # 75 : 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame # 76 : 0x0000000100148fdf Python`call_function + 441 frame # 77 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 78 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 79 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 80 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 81 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 82 : 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame # 83 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 84 : 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame # 85 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 86 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 87 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 88 : 0x00000001000f6fce Python`slot_tp_call + 50 frame # 89 : 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame # 90 : 0x0000000100148fdf Python`call_function + 441 frame # 91 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 92 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 93 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 94 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 95 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 96 : 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame # 97 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 98 : 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame # 99 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 100 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 101 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 102 : 0x00000001000f6fce Python`slot_tp_call + 50 frame # 103 : 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame # 104 : 0x0000000100148fdf Python`call_function + 441 frame # 105 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 106 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 107 : 0x0000000100148fe6 Python`call_function + 448 frame # 108 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 109 : 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame # 110 : 0x0000000100148fe6 Python`call_function + 448 frame # 111 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 112 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 113 : 0x0000000100140b4f Python`PyEval_EvalCodeEx + 52 frame # 114 : 0x00000001000ccc1b Python`function_call + 338 frame # 115 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 116 : 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame # 117 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 118 : 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame # 119 : 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame # 120 : 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame # 121 : 0x00000001000ac93a Python`PyObject_Call + 101 frame # 122 : 0x00000001000f78fa Python`slot_tp_init + 57 frame # 123 : 0x00000001000f4846 Python`type_call + 184 frame # 124 : 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame # 125 : 0x0000000100148fdf Python`call_function + 441 frame # 126 : 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame # 127 : 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame # 128 : 0x0000000100140b15 Python`PyEval_EvalCode + 43 frame # 129 : 0x000000010016825a Python`run_mod + 54 frame # 130 : 0x000000010016853a Python`PyRun_FileExFlags + 180 frame # 131 : 0x0000000100167ac7 Python`PyRun_SimpleFileExFlags + 280 frame # 132 : 0x000000010017b878 Python`Py_Main + 3268 frame # 133 : 0x0000000100001e1d python`___lldb_unnamed_symbol1$$python + 227 frame # 134 : 0x00007fff7e325115 libdyld.dylib`start + 1
          Hide
          Ellis.Breen Ellis Breen added a comment - - edited

          Crash is actually here:

          // @param key is not null-terminated.
          Value& Value::resolveReference(char const* key, char const* cend)
          {
            JSON_ASSERT_MESSAGE(
                type_ == nullValue || type_ == objectValue,
                "in Json::Value::resolveReference(key, end): requires objectValue");
            if (type_ == nullValue)
              *this = Value(objectValue);
            CZString actualKey(
                key, static_cast<unsigned>(cend-key), CZString::duplicateOnCopy);
            ObjectValues::iterator it = value_.map_->lower_bound(actualKey);
            if (it != value_.map_->end() && (*it).first == actualKey)
              return (*it).second;
           
            ObjectValues::value_type defaultValue(actualKey, nullRef);
            it = value_.map_->insert(it, defaultValue);
            Value& value = (*it).second;
            return value;
          }[code]

           

          But caused by seemingly uninitialised span here in span.cc:

          LIBCOUCHBASE_API
          void lcbtrace_span_add_tag_uint64(lcbtrace_SPAN *span, const char *name, uint64_t value)
          {
              span->add_tag(name, (Json::Value::UInt64)value);
          }

          Caused by this in tracing-internal.h:

          #define LCBTRACE_KV_FINISH(pipeline, request, response)                                                                \
              do {                                                                                                               \
                  lcbtrace_SPAN *span = MCREQ_PKT_RDATA(request)->span;                                                          \
                  if (span) {                                                                                                    \
                      lcbtrace_span_add_tag_uint64(span, LCBTRACE_TAG_PEER_LATENCY, (response)->duration());                     \
                      lcb::Server *server = static_cast< lcb::Server * >(pipeline);                                              \
                      const lcb_host_t &remote = server->get_host();                                                             \
                      std::string hh;                                                                                            \
                      if (remote.ipv6) {                                                                                         \
                          hh.append("[").append(remote.host).append("]:").append(remote.port);                                   \
                      } else {                                                                                                   \
                          hh.append(remote.host).append(":").append(remote.port);                                                \
                      }                                                                                                          \
                      lcbtrace_span_add_tag_str(span, LCBTRACE_TAG_PEER_ADDRESS, hh.c_str());                                    \
                      lcbio_CTX *ctx = server->connctx;                                                                          \
                      if (ctx) {                                                                                                 \
                          lcbtrace_span_add_tag_str(span, LCBTRACE_TAG_LOCAL_ADDRESS,                                            \
                                                    lcbio__inet_ntop(&ctx->sock->info->sa_local).c_str());                       \
                      }                                                                                                          \
                      lcbtrace_span_finish(span, LCBTRACE_NOW);                                                                  \
                  }                                                                                                              \
              } while (0);

          Show
          Ellis.Breen Ellis Breen added a comment - - edited Crash is actually here: // @param key is not null-terminated. Value& Value::resolveReference( char const * key, char const * cend) { JSON_ASSERT_MESSAGE( type_ == nullValue || type_ == objectValue, "in Json::Value::resolveReference(key, end): requires objectValue" ); if (type_ == nullValue) * this = Value(objectValue); CZString actualKey( key, static_cast<unsigned>(cend-key), CZString::duplicateOnCopy); ObjectValues::iterator it = value_.map_->lower_bound(actualKey); if (it != value_.map_->end() && (*it).first == actualKey) return (*it).second;   ObjectValues::value_type defaultValue(actualKey, nullRef); it = value_.map_->insert(it, defaultValue); Value& value = (*it).second; return value; }[code]   But caused by seemingly uninitialised span here in span.cc: LIBCOUCHBASE_API void lcbtrace_span_add_tag_uint64(lcbtrace_SPAN *span, const char *name, uint64_t value) { span->add_tag(name, (Json::Value::UInt64)value); } Caused by this in tracing-internal.h: #define LCBTRACE_KV_FINISH(pipeline, request, response) \ do { \ lcbtrace_SPAN *span = MCREQ_PKT_RDATA(request)->span; \ if (span) { \ lcbtrace_span_add_tag_uint64(span, LCBTRACE_TAG_PEER_LATENCY, (response)->duration()); \ lcb::Server *server = static_cast< lcb::Server * >(pipeline); \ const lcb_host_t &remote = server->get_host(); \ std::string hh; \ if (remote.ipv6) { \ hh.append( "[" ).append(remote.host).append( "]:" ).append(remote.port); \ } else { \ hh.append(remote.host).append( ":" ).append(remote.port); \ } \ lcbtrace_span_add_tag_str(span, LCBTRACE_TAG_PEER_ADDRESS, hh.c_str()); \ lcbio_CTX *ctx = server->connctx; \ if (ctx) { \ lcbtrace_span_add_tag_str(span, LCBTRACE_TAG_LOCAL_ADDRESS, \ lcbio__inet_ntop(&ctx->sock->info->sa_local).c_str()); \ } \ lcbtrace_span_finish(span, LCBTRACE_NOW); \ } \ } while ( 0 );
          Hide
          Ellis.Breen Ellis Breen added a comment - - edited

          Some variables from Frame 4 where it calls the above macro:

           

          // code placeholder
          (lldb) frame select 4
          frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748
          745 }
          746 w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL;
          747 handle_mutation_token(root, response, request, &w.mt);
          -> 748 LCBTRACE_KV_FINISH(pipeline, request, response);
          749 TRACE_STORE_END(root, request, response, &w.resp);
          750 if (request->flags & MCREQ_F_REQEXT) {
          751 request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp);
          (lldb) frame variable
          (mc_PIPELINE *) pipeline = 0x0000000104400270
          (mc_PACKET *) request = 0x0000000107001800
          (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0
          (lcb_error_t) immerr = LCB_SUCCESS
          (lcb_t) root = 0x00000001004cba70
          (ResponsePack<(anonymous struct)>) w = {
          resp = (cookie = 0x00000001056a7ed0, key = 0x0000000107002420, nkey = 3, cas = 1519328678316802048, rc = LCB_SUCCESS, version = 0, rflags = 9, op = LCB_SET)
          mt = (uuid_ = 0, seqno_ = 0, vbid_ = 0)
          value = 0x0000000000000000
          nvalue = 0
          err_ref = 0x0000000000000000
          err_ctx = 0x0000000000000000
          }
          (uint8_t) opcode = '\x01'
          (lcbtrace_SPAN *) span = 0x565f434f44425553
          (lcb::Server *) server = 0x0202024002020240
          (const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t", port = "ort argument into a sequence of 5-tuples that contain\n all the necessary arguments for creating a socket connected to that service.\n host is a domain name, a string representation of an IPv4/v6 address or\n None. port is a string service name such as 'http', a numeric port number or\n None. By passing None as the value of host and port, you can pass NULL to\n the underlying C API.\n\n The family, type and proto arguments can be optionally specified in order to\n narrow the list of addresses returned. Passing zero as a value for each of\n these arguments selects the full range of results.\n ", ipv6 = 1819309428)
          (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0"
          (lcbio_CTX *) ctx = 0x000000017e4ce38

          Show
          Ellis.Breen Ellis Breen added a comment - - edited Some variables from Frame 4 where it calls the above macro:   // code placeholder (lldb) frame select 4 frame # 4 : 0x00000001050859a7 libcouchbase. 2 .dylib`H_store(pipeline= 0x0000000104400270 , request= 0x0000000107001800 , response= 0x00007ffeefbfb2d0 , immerr=LCB_SUCCESS) at handler.cc: 748 745 } 746 w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL; 747 handle_mutation_token(root, response, request, &w.mt); -> 748 LCBTRACE_KV_FINISH(pipeline, request, response); 749 TRACE_STORE_END(root, request, response, &w.resp); 750 if (request->flags & MCREQ_F_REQEXT) { 751 request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp); (lldb) frame variable (mc_PIPELINE *) pipeline = 0x0000000104400270 (mc_PACKET *) request = 0x0000000107001800 (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0 (lcb_error_t) immerr = LCB_SUCCESS (lcb_t) root = 0x00000001004cba70 (ResponsePack<(anonymous struct)>) w = { resp = (cookie = 0x00000001056a7ed0 , key = 0x0000000107002420 , nkey = 3 , cas = 1519328678316802048 , rc = LCB_SUCCESS, version = 0 , rflags = 9 , op = LCB_SET) mt = (uuid_ = 0 , seqno_ = 0 , vbid_ = 0 ) value = 0x0000000000000000 nvalue = 0 err_ref = 0x0000000000000000 err_ctx = 0x0000000000000000 } (uint8_t) opcode = '\x01' (lcbtrace_SPAN *) span = 0x565f434f44425553 (lcb::Server *) server = 0x0202024002020240 ( const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t" , port = "ort argument into a sequence of 5-tuples that contain\n all the necessary arguments for creating a socket connected to that service.\n host is a domain name, a string representation of an IPv4/v6 address or\n None. port is a string service name such as 'http', a numeric port number or\n None. By passing None as the value of host and port, you can pass NULL to\n the underlying C API.\n\n The family, type and proto arguments can be optionally specified in order to\n narrow the list of addresses returned. Passing zero as a value for each of\n these arguments selects the full range of results.\n " , ipv6 = 1819309428 ) (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0" (lcbio_CTX *) ctx = 0x000000017e4ce38
          Hide
          Ellis.Breen Ellis Breen added a comment - - edited

          Some more exploration - let me know if you need any more clues.

            
          (lldb) frame select 4
          frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748
             745 	    }
             746 	    w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL;
             747 	    handle_mutation_token(root, response, request, &w.mt);
          -> 748 	    LCBTRACE_KV_FINISH(pipeline, request, response);
             749 	    TRACE_STORE_END(root, request, response, &w.resp);
             750 	    if (request->flags & MCREQ_F_REQEXT) {
             751 	        request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp);
          (lldb) frame variable
          (mc_PIPELINE *) pipeline = 0x0000000104400270
          (mc_PACKET *) request = 0x0000000107001800
          (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0
          (lcb_error_t) immerr = LCB_SUCCESS
          (lcb_t) root = 0x00000001004cba70
          (ResponsePack<(anonymous struct)>) w = {
            resp = (cookie = 0x00000001056a7ed0, key = 0x0000000107002420, nkey = 3, cas = 1519328678316802048, rc = LCB_SUCCESS, version = 0, rflags = 9, op = LCB_SET)
            mt = (uuid_ = 0, seqno_ = 0, vbid_ = 0)
            value = 0x0000000000000000
            nvalue = 0
            err_ref = 0x0000000000000000
            err_ctx = 0x0000000000000000
          }
          (uint8_t) opcode = '\x01'
          (lcbtrace_SPAN *) span = 0x565f434f44425553
          (lcb::Server *) server = 0x0202024002020240
          (const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t", port = "ort argument into a sequence of 5-tuples that contain\n    all the necessary arguments for creating a socket connected to that service.\n    host is a domain name, a string representation of an IPv4/v6 address or\n    None. port is a string service name such as 'http', a numeric port number or\n    None. By passing None as the value of host and port, you can pass NULL to\n    the underlying C API.\n\n    The family, type and proto arguments can be optionally specified in order to\n    narrow the list of addresses returned. Passing zero as a value for each of\n    these arguments selects the full range of results.\n    ", ipv6 = 1819309428)
          (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0"
          (lcbio_CTX *) ctx = 0x000000017e4ce384
          (lldb) bt
          * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
              frame #0: 0x000000010513a857 libcouchbase.2.dylib`Json::Value::resolveReference(this=0x565f434f4442558b, key="peer.latency", cend="") at lcb-jsoncpp.cpp:3161
              frame #1: 0x000000010512b0ab libcouchbase.2.dylib`Json::Value::operator[](this=0x565f434f4442558b, key="peer.latency") at lcb-jsoncpp.cpp:3210
              frame #2: 0x000000010510c93d libcouchbase.2.dylib`void lcb::trace::Span::add_tag<unsigned long long>(this=0x565f434f44425553, name="peer.latency", value=0) at span.cc:201
              frame #3: 0x000000010510c8f5 libcouchbase.2.dylib`::lcbtrace_span_add_tag_uint64(span=0x565f434f44425553, name="peer.latency", value=0) at span.cc:49
            * frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748
              frame #5: 0x000000010508447d libcouchbase.2.dylib`::mcreq_dispatch_response(pipeline=0x0000000104400270, req=0x0000000107001800, res=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:963
              frame #6: 0x00000001050efb9d libcouchbase.2.dylib`lcb::Server::try_read(this=0x0000000104400270, ctx=0x00000001044011a0, ior=0x00000001044011e8) at mcserver.cc:399
              frame #7: 0x00000001050f1884 libcouchbase.2.dylib`on_read(ctx=0x00000001044011a0, (null)=24) at mcserver.cc:440
              frame #8: 0x000000010501cac5 libcouchbase.2.dylib`invoke_read_cb(ctx=0x00000001044011a0, nb=24) at ctx.c:274
              frame #9: 0x000000010501c729 libcouchbase.2.dylib`E_handler(sock=10, which=2, arg=0x00000001044011a0) at ctx.c:303
              frame #10: 0x000000010572f9d7 libevent_core-2.1.6.dylib`event_process_active_single_queue + 1003
              frame #11: 0x000000010572cd39 libevent_core-2.1.6.dylib`event_base_loop + 1170
              frame #12: 0x00000001043f971e libcouchbase_libevent.dylib`lcb_io_run_event_loop(iops=0x0000000100476fc0) at plugin-libevent.c:202
              frame #13: 0x000000010510c640 libcouchbase.2.dylib`::lcb_wait3(instance=0x00000001004cba70, flags=LCB_WAIT_NOCHECK) at wait.cc:140
              frame #14: 0x0000000104752cb8 _libcouchbase.cpython-36m-darwin.so`pycbc_oputil_wait_common(self=0x000000010462e750) at oputil.c:461
              frame #15: 0x0000000104752b2a _libcouchbase.cpython-36m-darwin.so`pycbc_common_vars_wait(cv=0x00007ffeefbfb6c8, self=0x000000010462e750) at oputil.c:61
              frame #16: 0x000000010474cfa4 _libcouchbase.cpython-36m-darwin.so`set_common(self=0x000000010462e750, args=0x000000010575ff88, kwargs=0x000000010570afc0, operation=3, argopts=1) at store.c:341
              frame #17: 0x000000010474d120 _libcouchbase.cpython-36m-darwin.so`pycbc_Bucket_upsert(self=0x000000010462e750, args=0x000000010575ff88, kwargs=0x000000010570afc0) at store.c:363
              frame #18: 0x00000001000e47ea Python`_PyCFunction_FastCallDict + 463
              frame #19: 0x00000001000acacc Python`_PyObject_FastCallDict + 215
              frame #20: 0x00000001000bf7f6 Python`methoddescr_call + 114
              frame #21: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
              frame #22: 0x00000001000ace3d Python`_PyObject_FastCallKeywords + 97
              frame #23: 0x0000000100148fdf Python`call_function + 441
              frame #24: 0x0000000100141f64 Python`_PyEval_EvalFrameDefault + 5082
              frame #25: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #26: 0x0000000100149ecb Python`fast_function + 227
              frame #27: 0x0000000100148fe6 Python`call_function + 448
              frame #28: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #29: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #30: 0x0000000100148fe6 Python`call_function + 448
              frame #31: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #32: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #33: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
              frame #34: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #35: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #36: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #37: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
              frame #38: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #39: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
              frame #40: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #41: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #42: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #43: 0x00000001000f6fce Python`slot_tp_call + 50
              frame #44: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
              frame #45: 0x0000000100148fdf Python`call_function + 441
              frame #46: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #47: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #48: 0x0000000100148fe6 Python`call_function + 448
              frame #49: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #50: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #51: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #52: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #53: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #54: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
              frame #55: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #56: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
              frame #57: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #58: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #59: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #60: 0x00000001000f6fce Python`slot_tp_call + 50
              frame #61: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
              frame #62: 0x0000000100148fdf Python`call_function + 441
              frame #63: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #64: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #65: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #66: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #67: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #68: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
              frame #69: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #70: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
              frame #71: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #72: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #73: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #74: 0x00000001000f6fce Python`slot_tp_call + 50
              frame #75: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
              frame #76: 0x0000000100148fdf Python`call_function + 441
              frame #77: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #78: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #79: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #80: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #81: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #82: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
              frame #83: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #84: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
              frame #85: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #86: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #87: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #88: 0x00000001000f6fce Python`slot_tp_call + 50
              frame #89: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
              frame #90: 0x0000000100148fdf Python`call_function + 441
              frame #91: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #92: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #93: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #94: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #95: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #96: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
              frame #97: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #98: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
              frame #99: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #100: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #101: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #102: 0x00000001000f6fce Python`slot_tp_call + 50
              frame #103: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
              frame #104: 0x0000000100148fdf Python`call_function + 441
              frame #105: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #106: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #107: 0x0000000100148fe6 Python`call_function + 448
              frame #108: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #109: 0x000000010014a166 Python`_PyFunction_FastCall + 121
              frame #110: 0x0000000100148fe6 Python`call_function + 448
              frame #111: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #112: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #113: 0x0000000100140b4f Python`PyEval_EvalCodeEx + 52
              frame #114: 0x00000001000ccc1b Python`function_call + 338
              frame #115: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #116: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527
              frame #117: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #118: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444
              frame #119: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196
              frame #120: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156
              frame #121: 0x00000001000ac93a Python`PyObject_Call + 101
              frame #122: 0x00000001000f78fa Python`slot_tp_init + 57
              frame #123: 0x00000001000f4846 Python`type_call + 184
              frame #124: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143
              frame #125: 0x0000000100148fdf Python`call_function + 441
              frame #126: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930
              frame #127: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829
              frame #128: 0x0000000100140b15 Python`PyEval_EvalCode + 43
              frame #129: 0x000000010016825a Python`run_mod + 54
              frame #130: 0x000000010016853a Python`PyRun_FileExFlags + 180
              frame #131: 0x0000000100167ac7 Python`PyRun_SimpleFileExFlags + 280
              frame #132: 0x000000010017b878 Python`Py_Main + 3268
              frame #133: 0x0000000100001e1d python`___lldb_unnamed_symbol1$$python + 227
              frame #134: 0x00007fff7e325115 libdyld.dylib`start + 1
          (lldb) frame select 4
          frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748
             745 	    }
             746 	    w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL;
             747 	    handle_mutation_token(root, response, request, &w.mt);
          -> 748 	    LCBTRACE_KV_FINISH(pipeline, request, response);
             749 	    TRACE_STORE_END(root, request, response, &w.resp);
             750 	    if (request->flags & MCREQ_F_REQEXT) {
             751 	        request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp);
          (lldb) frame variable
          (mc_PIPELINE *) pipeline = 0x0000000104400270
          (mc_PACKET *) request = 0x0000000107001800
          (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0
          (lcb_error_t) immerr = LCB_SUCCESS
          (lcb_t) root = 0x00000001004cba70
          (ResponsePack<(anonymous struct)>) w = {
            resp = (cookie = 0x00000001056a7ed0, key = 0x0000000107002420, nkey = 3, cas = 1519328678316802048, rc = LCB_SUCCESS, version = 0, rflags = 9, op = LCB_SET)
            mt = (uuid_ = 0, seqno_ = 0, vbid_ = 0)
            value = 0x0000000000000000
            nvalue = 0
            err_ref = 0x0000000000000000
            err_ctx = 0x0000000000000000
          }
          (uint8_t) opcode = '\x01'
          (lcbtrace_SPAN *) span = 0x565f434f44425553
          (lcb::Server *) server = 0x0202024002020240
          (const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t", port = "ort argument into a sequence of 5-tuples that contain\n    all the necessary arguments for creating a socket connected to that service.\n    host is a domain name, a string representation of an IPv4/v6 address or\n    None. port is a string service name such as 'http', a numeric port number or\n    None. By passing None as the value of host and port, you can pass NULL to\n    the underlying C API.\n\n    The family, type and proto arguments can be optionally specified in order to\n    narrow the list of addresses returned. Passing zero as a value for each of\n    these arguments selects the full range of results.\n    ", ipv6 = 1819309428)
          (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0"
          (lcbio_CTX *) ctx = 0x000000017e4ce384
          (lldb) eval
          error: 'eval' is not a valid command.
          error: Unrecognized command 'eval'.
          (lldb) print
          Enter expressions, then terminate with an empty line to evaluate:
          (lldb) print request->flags & MCREQ_F_REQEXT
          error: use of undeclared identifier 'MCREQ_F_REQEXT'
          (lldb) print request->flags & (1<<4)
          (int) $0 = 0
          (lldb) print (&(pkt)->u_rdate.reqdata))
          error: use of undeclared identifier 'pkt'
          (lldb) print (&(request)->u_rdate.reqdata))
          error: no member named 'u_rdate' in 'mc_packet_st'
          (lldb) print (&(request)->u_rdata.reqdata))
          (mc_REQDATA *) $1 = 0x0000000107001828
            Fix-it applied, fixed expression was:
              (&(request)->u_rdata.reqdata)
          (lldb) print *(&(request)->u_rdata.reqdata))
          (mc_REQDATA) $2 = {
            cookie = 0x00000001056a7ed0
            start = 86400420847170
            dispatch = 86400421951753
            span = 0x565f434f44425553
          }
            Fix-it applied, fixed expression was:
              *(&(request)->u_rdata.reqdata)
          (lldb) print (&(request)->u_rdata.reqdata)
          (mc_REQDATA *) $3 = 0x0000000107001828
          (lldb) print *(&(request)->u_rdata.reqdata)
          (mc_REQDATA) $4 = {
            cookie = 0x00000001056a7ed0
            start = 86400420847170
            dispatch = 86400421951753
            span = 0x565f434f44425553
          }
          (lldb) frame 3
          invalid command 'frame 3'.
          (lldb) frame select 3
          frame #3: 0x000000010510c8f5 libcouchbase.2.dylib`::lcbtrace_span_add_tag_uint64(span=0x565f434f44425553, name="peer.latency", value=0) at span.cc:49
             46  	LIBCOUCHBASE_API
             47  	void lcbtrace_span_add_tag_uint64(lcbtrace_SPAN *span, const char *name, uint64_t value)
             48  	{
          -> 49  	    span->add_tag(name, (Json::Value::UInt64)value);
             50  	}
             51
             52  	LIBCOUCHBASE_API
          (lldb) frame 2
          invalid command 'frame 2'.
          (lldb) frame select 2
          frame #2: 0x000000010510c93d libcouchbase.2.dylib`void lcb::trace::Span::add_tag<unsigned long long>(this=0x565f434f44425553, name="peer.latency", value=0) at span.cc:201
             198
             199 	template < typename T > void Span::add_tag(const char *name, T value)
             200 	{
          -> 201 	    tags[name] = value;
             202 	}
          (lldb) frame variable
          (lcb::trace::Span *) this = 0x565f434f44425553
          (const char *) name = 0x000000010518688d "peer.latency"
          (unsigned long long) value = 0
          (lldb) print *this
          error: Couldn't apply expression side effects : Couldn't dematerialize a result variable: couldn't read its memory
          (lldb) frame select 3
          frame #3: 0x000000010510c8f5 libcouchbase.2.dylib`::lcbtrace_span_add_tag_uint64(span=0x565f434f44425553, name="peer.latency", value=0) at span.cc:49
             46  	LIBCOUCHBASE_API
             47  	void lcbtrace_span_add_tag_uint64(lcbtrace_SPAN *span, const char *name, uint64_t value)
             48  	{
          -> 49  	    span->add_tag(name, (Json::Value::UInt64)value);
             50  	}
             51
             52  	LIBCOUCHBASE_API
          (lldb) frame variable
          (lcbtrace_SPAN *) span = 0x565f434f44425553
          (const char *) name = 0x000000010518688d "peer.latency"
          (uint64_t) value = 0
          (lldb) frame select 4
          frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748
             745 	    }
             746 	    w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL;
             747 	    handle_mutation_token(root, response, request, &w.mt);
          -> 748 	    LCBTRACE_KV_FINISH(pipeline, request, response);
             749 	    TRACE_STORE_END(root, request, response, &w.resp);
             750 	    if (request->flags & MCREQ_F_REQEXT) {
             751 	        request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp);
          (lldb) frame variable
          (mc_PIPELINE *) pipeline = 0x0000000104400270
          (mc_PACKET *) request = 0x0000000107001800
          (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0
          (lcb_error_t) immerr = LCB_SUCCESS
          (lcb_t) root = 0x00000001004cba70
          (ResponsePack<(anonymous struct)>) w = {
            resp = (cookie = 0x00000001056a7ed0, key = 0x0000000107002420, nkey = 3, cas = 1519328678316802048, rc = LCB_SUCCESS, version = 0, rflags = 9, op = LCB_SET)
            mt = (uuid_ = 0, seqno_ = 0, vbid_ = 0)
            value = 0x0000000000000000
            nvalue = 0
            err_ref = 0x0000000000000000
            err_ctx = 0x0000000000000000
          }
          (uint8_t) opcode = '\x01'
          (lcbtrace_SPAN *) span = 0x565f434f44425553
          (lcb::Server *) server = 0x0202024002020240
          (const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t", port = "ort argument into a sequence of 5-tuples that contain\n    all the necessary arguments for creating a socket connected to that service.\n    host is a domain name, a string representation of an IPv4/v6 address or\n    None. port is a string service name such as 'http', a numeric port number or\n    None. By passing None as the value of host and port, you can pass NULL to\n    the underlying C API.\n\n    The family, type and proto arguments can be optionally specified in order to\n    narrow the list of addresses returned. Passing zero as a value for each of\n    these arguments selects the full range of results.\n    ", ipv6 = 1819309428)
          (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0"
          (lcbio_CTX *) ctx = 0x000000017e4ce384
          (lldb)

          Show
          Ellis.Breen Ellis Breen added a comment - - edited Some more exploration - let me know if you need any more clues. (lldb) frame select 4 frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748 745 } 746 w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL; 747 handle_mutation_token(root, response, request, &w.mt); -> 748 LCBTRACE_KV_FINISH(pipeline, request, response); 749 TRACE_STORE_END(root, request, response, &w.resp); 750 if (request->flags & MCREQ_F_REQEXT) { 751 request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp); (lldb) frame variable (mc_PIPELINE *) pipeline = 0x0000000104400270 (mc_PACKET *) request = 0x0000000107001800 (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0 (lcb_error_t) immerr = LCB_SUCCESS (lcb_t) root = 0x00000001004cba70 (ResponsePack<(anonymous struct)>) w = { resp = (cookie = 0x00000001056a7ed0, key = 0x0000000107002420, nkey = 3, cas = 1519328678316802048, rc = LCB_SUCCESS, version = 0, rflags = 9, op = LCB_SET) mt = (uuid_ = 0, seqno_ = 0, vbid_ = 0) value = 0x0000000000000000 nvalue = 0 err_ref = 0x0000000000000000 err_ctx = 0x0000000000000000 } (uint8_t) opcode = '\x01' (lcbtrace_SPAN *) span = 0x565f434f44425553 (lcb::Server *) server = 0x0202024002020240 (const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t", port = "ort argument into a sequence of 5-tuples that contain\n all the necessary arguments for creating a socket connected to that service.\n host is a domain name, a string representation of an IPv4/v6 address or\n None. port is a string service name such as 'http', a numeric port number or\n None. By passing None as the value of host and port, you can pass NULL to\n the underlying C API.\n\n The family, type and proto arguments can be optionally specified in order to\n narrow the list of addresses returned. Passing zero as a value for each of\n these arguments selects the full range of results.\n ", ipv6 = 1819309428) (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0" (lcbio_CTX *) ctx = 0x000000017e4ce384 (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT) frame #0: 0x000000010513a857 libcouchbase.2.dylib`Json::Value::resolveReference(this=0x565f434f4442558b, key="peer.latency", cend="") at lcb-jsoncpp.cpp:3161 frame #1: 0x000000010512b0ab libcouchbase.2.dylib`Json::Value::operator[](this=0x565f434f4442558b, key="peer.latency") at lcb-jsoncpp.cpp:3210 frame #2: 0x000000010510c93d libcouchbase.2.dylib`void lcb::trace::Span::add_tag<unsigned long long>(this=0x565f434f44425553, name="peer.latency", value=0) at span.cc:201 frame #3: 0x000000010510c8f5 libcouchbase.2.dylib`::lcbtrace_span_add_tag_uint64(span=0x565f434f44425553, name="peer.latency", value=0) at span.cc:49 * frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748 frame #5: 0x000000010508447d libcouchbase.2.dylib`::mcreq_dispatch_response(pipeline=0x0000000104400270, req=0x0000000107001800, res=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:963 frame #6: 0x00000001050efb9d libcouchbase.2.dylib`lcb::Server::try_read(this=0x0000000104400270, ctx=0x00000001044011a0, ior=0x00000001044011e8) at mcserver.cc:399 frame #7: 0x00000001050f1884 libcouchbase.2.dylib`on_read(ctx=0x00000001044011a0, (null)=24) at mcserver.cc:440 frame #8: 0x000000010501cac5 libcouchbase.2.dylib`invoke_read_cb(ctx=0x00000001044011a0, nb=24) at ctx.c:274 frame #9: 0x000000010501c729 libcouchbase.2.dylib`E_handler(sock=10, which=2, arg=0x00000001044011a0) at ctx.c:303 frame #10: 0x000000010572f9d7 libevent_core-2.1.6.dylib`event_process_active_single_queue + 1003 frame #11: 0x000000010572cd39 libevent_core-2.1.6.dylib`event_base_loop + 1170 frame #12: 0x00000001043f971e libcouchbase_libevent.dylib`lcb_io_run_event_loop(iops=0x0000000100476fc0) at plugin-libevent.c:202 frame #13: 0x000000010510c640 libcouchbase.2.dylib`::lcb_wait3(instance=0x00000001004cba70, flags=LCB_WAIT_NOCHECK) at wait.cc:140 frame #14: 0x0000000104752cb8 _libcouchbase.cpython-36m-darwin.so`pycbc_oputil_wait_common(self=0x000000010462e750) at oputil.c:461 frame #15: 0x0000000104752b2a _libcouchbase.cpython-36m-darwin.so`pycbc_common_vars_wait(cv=0x00007ffeefbfb6c8, self=0x000000010462e750) at oputil.c:61 frame #16: 0x000000010474cfa4 _libcouchbase.cpython-36m-darwin.so`set_common(self=0x000000010462e750, args=0x000000010575ff88, kwargs=0x000000010570afc0, operation=3, argopts=1) at store.c:341 frame #17: 0x000000010474d120 _libcouchbase.cpython-36m-darwin.so`pycbc_Bucket_upsert(self=0x000000010462e750, args=0x000000010575ff88, kwargs=0x000000010570afc0) at store.c:363 frame #18: 0x00000001000e47ea Python`_PyCFunction_FastCallDict + 463 frame #19: 0x00000001000acacc Python`_PyObject_FastCallDict + 215 frame #20: 0x00000001000bf7f6 Python`methoddescr_call + 114 frame #21: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame #22: 0x00000001000ace3d Python`_PyObject_FastCallKeywords + 97 frame #23: 0x0000000100148fdf Python`call_function + 441 frame #24: 0x0000000100141f64 Python`_PyEval_EvalFrameDefault + 5082 frame #25: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #26: 0x0000000100149ecb Python`fast_function + 227 frame #27: 0x0000000100148fe6 Python`call_function + 448 frame #28: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #29: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #30: 0x0000000100148fe6 Python`call_function + 448 frame #31: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #32: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #33: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame #34: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #35: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #36: 0x00000001000ac93a Python`PyObject_Call + 101 frame #37: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame #38: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #39: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame #40: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #41: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #42: 0x00000001000ac93a Python`PyObject_Call + 101 frame #43: 0x00000001000f6fce Python`slot_tp_call + 50 frame #44: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame #45: 0x0000000100148fdf Python`call_function + 441 frame #46: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #47: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #48: 0x0000000100148fe6 Python`call_function + 448 frame #49: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #50: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #51: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #52: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #53: 0x00000001000ac93a Python`PyObject_Call + 101 frame #54: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame #55: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #56: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame #57: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #58: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #59: 0x00000001000ac93a Python`PyObject_Call + 101 frame #60: 0x00000001000f6fce Python`slot_tp_call + 50 frame #61: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame #62: 0x0000000100148fdf Python`call_function + 441 frame #63: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #64: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #65: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #66: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #67: 0x00000001000ac93a Python`PyObject_Call + 101 frame #68: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame #69: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #70: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame #71: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #72: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #73: 0x00000001000ac93a Python`PyObject_Call + 101 frame #74: 0x00000001000f6fce Python`slot_tp_call + 50 frame #75: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame #76: 0x0000000100148fdf Python`call_function + 441 frame #77: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #78: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #79: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #80: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #81: 0x00000001000ac93a Python`PyObject_Call + 101 frame #82: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame #83: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #84: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame #85: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #86: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #87: 0x00000001000ac93a Python`PyObject_Call + 101 frame #88: 0x00000001000f6fce Python`slot_tp_call + 50 frame #89: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame #90: 0x0000000100148fdf Python`call_function + 441 frame #91: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #92: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #93: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #94: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #95: 0x00000001000ac93a Python`PyObject_Call + 101 frame #96: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame #97: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #98: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame #99: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #100: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #101: 0x00000001000ac93a Python`PyObject_Call + 101 frame #102: 0x00000001000f6fce Python`slot_tp_call + 50 frame #103: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame #104: 0x0000000100148fdf Python`call_function + 441 frame #105: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #106: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #107: 0x0000000100148fe6 Python`call_function + 448 frame #108: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #109: 0x000000010014a166 Python`_PyFunction_FastCall + 121 frame #110: 0x0000000100148fe6 Python`call_function + 448 frame #111: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #112: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #113: 0x0000000100140b4f Python`PyEval_EvalCodeEx + 52 frame #114: 0x00000001000ccc1b Python`function_call + 338 frame #115: 0x00000001000ac93a Python`PyObject_Call + 101 frame #116: 0x0000000100142121 Python`_PyEval_EvalFrameDefault + 5527 frame #117: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #118: 0x000000010014a096 Python`_PyFunction_FastCallDict + 444 frame #119: 0x00000001000acab9 Python`_PyObject_FastCallDict + 196 frame #120: 0x00000001000acbdc Python`_PyObject_Call_Prepend + 156 frame #121: 0x00000001000ac93a Python`PyObject_Call + 101 frame #122: 0x00000001000f78fa Python`slot_tp_init + 57 frame #123: 0x00000001000f4846 Python`type_call + 184 frame #124: 0x00000001000aca84 Python`_PyObject_FastCallDict + 143 frame #125: 0x0000000100148fdf Python`call_function + 441 frame #126: 0x0000000100141ecc Python`_PyEval_EvalFrameDefault + 4930 frame #127: 0x00000001001497df Python`_PyEval_EvalCodeWithName + 1829 frame #128: 0x0000000100140b15 Python`PyEval_EvalCode + 43 frame #129: 0x000000010016825a Python`run_mod + 54 frame #130: 0x000000010016853a Python`PyRun_FileExFlags + 180 frame #131: 0x0000000100167ac7 Python`PyRun_SimpleFileExFlags + 280 frame #132: 0x000000010017b878 Python`Py_Main + 3268 frame #133: 0x0000000100001e1d python`___lldb_unnamed_symbol1$$python + 227 frame #134: 0x00007fff7e325115 libdyld.dylib`start + 1 (lldb) frame select 4 frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748 745 } 746 w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL; 747 handle_mutation_token(root, response, request, &w.mt); -> 748 LCBTRACE_KV_FINISH(pipeline, request, response); 749 TRACE_STORE_END(root, request, response, &w.resp); 750 if (request->flags & MCREQ_F_REQEXT) { 751 request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp); (lldb) frame variable (mc_PIPELINE *) pipeline = 0x0000000104400270 (mc_PACKET *) request = 0x0000000107001800 (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0 (lcb_error_t) immerr = LCB_SUCCESS (lcb_t) root = 0x00000001004cba70 (ResponsePack<(anonymous struct)>) w = { resp = (cookie = 0x00000001056a7ed0, key = 0x0000000107002420, nkey = 3, cas = 1519328678316802048, rc = LCB_SUCCESS, version = 0, rflags = 9, op = LCB_SET) mt = (uuid_ = 0, seqno_ = 0, vbid_ = 0) value = 0x0000000000000000 nvalue = 0 err_ref = 0x0000000000000000 err_ctx = 0x0000000000000000 } (uint8_t) opcode = '\x01' (lcbtrace_SPAN *) span = 0x565f434f44425553 (lcb::Server *) server = 0x0202024002020240 (const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t", port = "ort argument into a sequence of 5-tuples that contain\n all the necessary arguments for creating a socket connected to that service.\n host is a domain name, a string representation of an IPv4/v6 address or\n None. port is a string service name such as 'http', a numeric port number or\n None. By passing None as the value of host and port, you can pass NULL to\n the underlying C API.\n\n The family, type and proto arguments can be optionally specified in order to\n narrow the list of addresses returned. Passing zero as a value for each of\n these arguments selects the full range of results.\n ", ipv6 = 1819309428) (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0" (lcbio_CTX *) ctx = 0x000000017e4ce384 (lldb) eval error: 'eval' is not a valid command. error: Unrecognized command 'eval'. (lldb) print Enter expressions, then terminate with an empty line to evaluate: (lldb) print request->flags & MCREQ_F_REQEXT error: use of undeclared identifier 'MCREQ_F_REQEXT' (lldb) print request->flags & (1<<4) (int) $0 = 0 (lldb) print (&(pkt)->u_rdate.reqdata)) error: use of undeclared identifier 'pkt' (lldb) print (&(request)->u_rdate.reqdata)) error: no member named 'u_rdate' in 'mc_packet_st' (lldb) print (&(request)->u_rdata.reqdata)) (mc_REQDATA *) $1 = 0x0000000107001828 Fix-it applied, fixed expression was: (&(request)->u_rdata.reqdata) (lldb) print *(&(request)->u_rdata.reqdata)) (mc_REQDATA) $2 = { cookie = 0x00000001056a7ed0 start = 86400420847170 dispatch = 86400421951753 span = 0x565f434f44425553 } Fix-it applied, fixed expression was: *(&(request)->u_rdata.reqdata) (lldb) print (&(request)->u_rdata.reqdata) (mc_REQDATA *) $3 = 0x0000000107001828 (lldb) print *(&(request)->u_rdata.reqdata) (mc_REQDATA) $4 = { cookie = 0x00000001056a7ed0 start = 86400420847170 dispatch = 86400421951753 span = 0x565f434f44425553 } (lldb) frame 3 invalid command 'frame 3'. (lldb) frame select 3 frame #3: 0x000000010510c8f5 libcouchbase.2.dylib`::lcbtrace_span_add_tag_uint64(span=0x565f434f44425553, name="peer.latency", value=0) at span.cc:49 46 LIBCOUCHBASE_API 47 void lcbtrace_span_add_tag_uint64(lcbtrace_SPAN *span, const char *name, uint64_t value) 48 { -> 49 span->add_tag(name, (Json::Value::UInt64)value); 50 } 51 52 LIBCOUCHBASE_API (lldb) frame 2 invalid command 'frame 2'. (lldb) frame select 2 frame #2: 0x000000010510c93d libcouchbase.2.dylib`void lcb::trace::Span::add_tag<unsigned long long>(this=0x565f434f44425553, name="peer.latency", value=0) at span.cc:201 198 199 template < typename T > void Span::add_tag(const char *name, T value) 200 { -> 201 tags[name] = value; 202 } (lldb) frame variable (lcb::trace::Span *) this = 0x565f434f44425553 (const char *) name = 0x000000010518688d "peer.latency" (unsigned long long) value = 0 (lldb) print *this error: Couldn't apply expression side effects : Couldn't dematerialize a result variable: couldn't read its memory (lldb) frame select 3 frame #3: 0x000000010510c8f5 libcouchbase.2.dylib`::lcbtrace_span_add_tag_uint64(span=0x565f434f44425553, name="peer.latency", value=0) at span.cc:49 46 LIBCOUCHBASE_API 47 void lcbtrace_span_add_tag_uint64(lcbtrace_SPAN *span, const char *name, uint64_t value) 48 { -> 49 span->add_tag(name, (Json::Value::UInt64)value); 50 } 51 52 LIBCOUCHBASE_API (lldb) frame variable (lcbtrace_SPAN *) span = 0x565f434f44425553 (const char *) name = 0x000000010518688d "peer.latency" (uint64_t) value = 0 (lldb) frame select 4 frame #4: 0x00000001050859a7 libcouchbase.2.dylib`H_store(pipeline=0x0000000104400270, request=0x0000000107001800, response=0x00007ffeefbfb2d0, immerr=LCB_SUCCESS) at handler.cc:748 745 } 746 w.resp.rflags |= LCB_RESP_F_EXTDATA | LCB_RESP_F_FINAL; 747 handle_mutation_token(root, response, request, &w.mt); -> 748 LCBTRACE_KV_FINISH(pipeline, request, response); 749 TRACE_STORE_END(root, request, response, &w.resp); 750 if (request->flags & MCREQ_F_REQEXT) { 751 request->u_rdata.exdata->procs->handler(pipeline, request, immerr, &w.resp); (lldb) frame variable (mc_PIPELINE *) pipeline = 0x0000000104400270 (mc_PACKET *) request = 0x0000000107001800 (lcb::MemcachedResponse *) response = 0x00007ffeefbfb2d0 (lcb_error_t) immerr = LCB_SUCCESS (lcb_t) root = 0x00000001004cba70 (ResponsePack<(anonymous struct)>) w = { resp = (cookie = 0x00000001056a7ed0, key = 0x0000000107002420, nkey = 3, cas = 1519328678316802048, rc = LCB_SUCCESS, version = 0, rflags = 9, op = LCB_SET) mt = (uuid_ = 0, seqno_ = 0, vbid_ = 0) value = 0x0000000000000000 nvalue = 0 err_ref = 0x0000000000000000 err_ctx = 0x0000000000000000 } (uint8_t) opcode = '\x01' (lcbtrace_SPAN *) span = 0x565f434f44425553 (lcb::Server *) server = 0x0202024002020240 (const lcb_host_t &) remote = 0x0000000100452610 (host = "*��t", port = "ort argument into a sequence of 5-tuples that contain\n all the necessary arguments for creating a socket connected to that service.\n host is a domain name, a string representation of an IPv4/v6 address or\n None. port is a string service name such as 'http', a numeric port number or\n None. By passing None as the value of host and port, you can pass NULL to\n the underlying C API.\n\n The family, type and proto arguments can be optionally specified in order to\n narrow the list of addresses returned. Passing zero as a value for each of\n these arguments selects the full range of results.\n ", ipv6 = 1819309428) (std::__1::string) hh = "�\x05\x01\0\0\0\x80`.\0e\0\0\0\x14�\x01\0\0\0\x10&E\0\x01\0\0\0@\x02\x02\x02@\x02\x02\x02SUBDOC_V@\x02\x02\x02@\x02\x02\x02\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0�����\0\0�@\x04\x01\0\0\x01�~j\x05\x01\0" (lcbio_CTX *) ctx = 0x000000017e4ce384 (lldb)
          Hide
          avsej Sergey Avseyev added a comment -

          Ellis Breen, thanks for report, could you also attach your code you use to reproduce it?

          Show
          avsej Sergey Avseyev added a comment - Ellis Breen , thanks for report, could you also attach your code you use to reproduce it?
          Hide
          Ellis.Breen Ellis Breen added a comment -

          Running the python3.6 version of

          nosetests couchbase.tests.misc_t

          a couple of times on the couchbase-python-client on the HEAD with a real local server seemed to trigger it - but it seems to be intermittent.

           

          Show
          Ellis.Breen Ellis Breen added a comment - Running the python3.6 version of nosetests couchbase.tests.misc_t a couple of times on the couchbase-python-client on the HEAD with a real local server seemed to trigger it - but it seems to be intermittent.  
          Hide
          build-team Couchbase Build Team added a comment -

          Build libcouchbase-2.8.5-131 contains libcouchbase commit 4bbf9fb7e09d81fbbb98a084ad1a2fab5eea3536 with commit message:
          CCBC-888: Tracing for HTTP services
          https://github.com/couchbase/libcouchbase/commit/4bbf9fb7e09d81fbbb98a084ad1a2fab5eea3536

          Show
          build-team Couchbase Build Team added a comment - Build libcouchbase-2.8.5-131 contains libcouchbase commit 4bbf9fb7e09d81fbbb98a084ad1a2fab5eea3536 with commit message: CCBC-888 : Tracing for HTTP services https://github.com/couchbase/libcouchbase/commit/4bbf9fb7e09d81fbbb98a084ad1a2fab5eea3536
          Hide
          build-team Couchbase Build Team added a comment -

          Build libcouchbase-2.8.5-137 contains libcouchbase commit a018a7b0798f7e22fc7503af9c7afdb2a1cef79d with commit message:
          CCBC-888: Add threshold logging tracer
          https://github.com/couchbase/libcouchbase/commit/a018a7b0798f7e22fc7503af9c7afdb2a1cef79d

          Show
          build-team Couchbase Build Team added a comment - Build libcouchbase-2.8.5-137 contains libcouchbase commit a018a7b0798f7e22fc7503af9c7afdb2a1cef79d with commit message: CCBC-888 : Add threshold logging tracer https://github.com/couchbase/libcouchbase/commit/a018a7b0798f7e22fc7503af9c7afdb2a1cef79d

            People

            • Assignee:
              avsej Sergey Avseyev
              Reporter:
              mike.goldsmith Michael Goldsmith
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.