Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-876

Null field in newsrv->authority leading to crash in lcb_wait3/compute_vb_list_diff

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.3
    • Fix Version/s: 2.8.4
    • Component/s: library
    • Labels:
      None

      Description

      bt
      * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
        * frame #0: 0x00007fff8ecc8330 libsystem_platform.dylib`_platform_strcmp + 176
          frame #1: 0x00000001052f54b2 libcouchbase.2.dylib`compute_vb_list_diff(from=0x000000010047cc90, to=0x0000000100436c30, out=0x0000000101952230) at vbucket.c:1051
          frame #2: 0x00000001052f5278 libcouchbase.2.dylib`lcbvb_compare(from=0x000000010047cc90, to=0x0000000100436c30) at vbucket.c:1076
          frame #3: 0x0000000105356969 libcouchbase.2.dylib`lcb::clconfig::Confmon::do_set_next(this=0x0000000100576000, new_config=0x0000000100419470, notify_miss=false) at confmon.cc:134
          frame #4: 0x0000000105354c63 libcouchbase.2.dylib`lcb::clconfig::Confmon::do_next_provider(this=0x0000000100576000) at confmon.cc:254
          frame #5: 0x0000000105359047 libcouchbase.2.dylib`lcb::io::Timer<lcb::clconfig::Confmon, &(lcb::clconfig::Confmon::do_next_provider())>::cb(arg=0x0000000100576000) at timer-cxx.h:80
          frame #6: 0x0000000105308c3d libcouchbase.2.dylib`timer_callback(sock=-1, which=1, arg=0x000000010053b8b0) at timer.c:45
          frame #7: 0x00000001056df9d7 libevent_core-2.1.6.dylib`event_process_active_single_queue + 1003
          frame #8: 0x00000001056dcd39 libevent_core-2.1.6.dylib`event_base_loop + 1170
          frame #9: 0x00000001056cb71e libcouchbase_libevent.dylib`lcb_io_run_event_loop(iops=0x0000000100579500) at plugin-libevent.c:202
          frame #10: 0x00000001053f5da0 libcouchbase.2.dylib`::lcb_wait3(instance=0x0000000100564a30, flags=LCB_WAIT_NOCHECK) at wait.cc:140
          frame #11: 0x00000001052c4df8 _libcouchbase.cpython-36m-darwin.so`pycbc_oputil_wait_common(self=0x00000001056af750) at oputil.c:461
          frame #12: 0x00000001052ca9b4 _libcouchbase.cpython-36m-darwin.so`pycbc_Bucket__http_request(self=0x00000001056af750, args=0x0000000100303048, kwargs=0x00000001056bf120) at http.c:265
          frame #13: 0x00000001000d17ea Python`_PyCFunction_FastCallDict + 463
          frame #14: 0x00000001000d18b5 Python`_PyCFunction_FastCallKeywords + 69
          frame #15: 0x000000010013600f Python`call_function + 489
          frame #16: 0x000000010012ef64 Python`_PyEval_EvalFrameDefault + 5082
          frame #17: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #18: 0x0000000100136ecb Python`fast_function + 227
          frame #19: 0x0000000100135fe6 Python`call_function + 448
          frame #20: 0x000000010012ef64 Python`_PyEval_EvalFrameDefault + 5082
          frame #21: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #22: 0x0000000100135fe6 Python`call_function + 448
          frame #23: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #24: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #25: 0x0000000100135fe6 Python`call_function + 448
          frame #26: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #27: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #28: 0x0000000100137096 Python`_PyFunction_FastCallDict + 444
          frame #29: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #30: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #31: 0x000000010009993a Python`PyObject_Call + 101
          frame #32: 0x000000010012f121 Python`_PyEval_EvalFrameDefault + 5527
          frame #33: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #34: 0x0000000100137096 Python`_PyFunction_FastCallDict + 444
          frame #35: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #36: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #37: 0x000000010009993a Python`PyObject_Call + 101
          frame #38: 0x00000001000e3fce Python`slot_tp_call + 50
          frame #39: 0x0000000100099a84 Python`_PyObject_FastCallDict + 143
          frame #40: 0x0000000100135fdf Python`call_function + 441
          frame #41: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #42: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #43: 0x0000000100135fe6 Python`call_function + 448
          frame #44: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #45: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #46: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #47: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #48: 0x000000010009993a Python`PyObject_Call + 101
          frame #49: 0x000000010012f121 Python`_PyEval_EvalFrameDefault + 5527
          frame #50: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #51: 0x0000000100137096 Python`_PyFunction_FastCallDict + 444
          frame #52: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #53: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #54: 0x000000010009993a Python`PyObject_Call + 101
          frame #55: 0x00000001000e3fce Python`slot_tp_call + 50
          frame #56: 0x0000000100099a84 Python`_PyObject_FastCallDict + 143
          frame #57: 0x0000000100135fdf Python`call_function + 441
          frame #58: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #59: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #60: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #61: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #62: 0x000000010009993a Python`PyObject_Call + 101
          frame #63: 0x000000010012f121 Python`_PyEval_EvalFrameDefault + 5527
          frame #64: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #65: 0x0000000100137096 Python`_PyFunction_FastCallDict + 444
          frame #66: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #67: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #68: 0x000000010009993a Python`PyObject_Call + 101
          frame #69: 0x00000001000e3fce Python`slot_tp_call + 50
          frame #70: 0x0000000100099a84 Python`_PyObject_FastCallDict + 143
          frame #71: 0x0000000100135fdf Python`call_function + 441
          frame #72: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #73: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #74: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #75: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #76: 0x000000010009993a Python`PyObject_Call + 101
          frame #77: 0x000000010012f121 Python`_PyEval_EvalFrameDefault + 5527
          frame #78: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #79: 0x0000000100137096 Python`_PyFunction_FastCallDict + 444
          frame #80: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #81: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #82: 0x000000010009993a Python`PyObject_Call + 101
          frame #83: 0x00000001000e3fce Python`slot_tp_call + 50
          frame #84: 0x0000000100099a84 Python`_PyObject_FastCallDict + 143
          frame #85: 0x0000000100135fdf Python`call_function + 441
          frame #86: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #87: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #88: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #89: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #90: 0x000000010009993a Python`PyObject_Call + 101
          frame #91: 0x000000010012f121 Python`_PyEval_EvalFrameDefault + 5527
          frame #92: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #93: 0x0000000100137096 Python`_PyFunction_FastCallDict + 444
          frame #94: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #95: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #96: 0x000000010009993a Python`PyObject_Call + 101
          frame #97: 0x00000001000e3fce Python`slot_tp_call + 50
          frame #98: 0x0000000100099a84 Python`_PyObject_FastCallDict + 143
          frame #99: 0x0000000100135fdf Python`call_function + 441
          frame #100: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #101: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #102: 0x0000000100135fe6 Python`call_function + 448
          frame #103: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #104: 0x0000000100137166 Python`_PyFunction_FastCall + 121
          frame #105: 0x0000000100135fe6 Python`call_function + 448
          frame #106: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #107: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #108: 0x000000010012db4f Python`PyEval_EvalCodeEx + 52
          frame #109: 0x00000001000b9c1b Python`function_call + 338
          frame #110: 0x000000010009993a Python`PyObject_Call + 101
          frame #111: 0x000000010012f121 Python`_PyEval_EvalFrameDefault + 5527
          frame #112: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #113: 0x0000000100137096 Python`_PyFunction_FastCallDict + 444
          frame #114: 0x0000000100099ab9 Python`_PyObject_FastCallDict + 196
          frame #115: 0x0000000100099bdc Python`_PyObject_Call_Prepend + 156
          frame #116: 0x000000010009993a Python`PyObject_Call + 101
          frame #117: 0x00000001000e48fa Python`slot_tp_init + 57
          frame #118: 0x00000001000e1846 Python`type_call + 184
          frame #119: 0x0000000100099a84 Python`_PyObject_FastCallDict + 143
          frame #120: 0x0000000100135fdf Python`call_function + 441
          frame #121: 0x000000010012eecc Python`_PyEval_EvalFrameDefault + 4930
          frame #122: 0x00000001001367df Python`_PyEval_EvalCodeWithName + 1829
          frame #123: 0x000000010012db15 Python`PyEval_EvalCode + 43
          frame #124: 0x000000010015525a Python`run_mod + 54
          frame #125: 0x000000010015553a Python`PyRun_FileExFlags + 180
          frame #126: 0x0000000100154ac7 Python`PyRun_SimpleFileExFlags + 280
          frame #127: 0x0000000100168878 Python`Py_Main + 3268
          frame #128: 0x0000000100001e1d python`___lldb_unnamed_symbol1$$python + 227
          frame #129: 0x00007fff8eab9235 libdyld.dylib`start + 1
          frame #130: 0x00007fff8eab9235 libdyld.dylib`start + 1
      (lldb) frame variable
      (lldb) frame select 1
      frame #1: 0x00000001052f54b2 libcouchbase.2.dylib`compute_vb_list_diff(from=0x000000010047cc90, to=0x0000000100436c30, out=0x0000000101952230) at vbucket.c:1051
         1048	        lcbvb_SERVER *newsrv = to->servers + ii;
         1049	        for (jj = 0; !found && jj < from->nsrv; jj++) {
         1050	            lcbvb_SERVER *oldsrv = from->servers + jj;
      -> 1051	            found |= (strcmp(newsrv->authority, oldsrv->authority) == 0);
         1052	        }
         1053	        if (!found) {
         1054	            char *infostr = malloc(strlen(newsrv->authority) + 128);
      (lldb) frame variable
      (lcbvb_CONFIG *) from = 0x000000010047cc90
      (lcbvb_CONFIG *) to = 0x0000000100436c30
      (char **) out = 0x0000000101952230
      (int) offset = 0
      (unsigned int) ii = 0
      (unsigned int) jj = 0
      (int) found = 0
      (lcbvb_SERVER *) newsrv = 0x000000010045af70
      (lcbvb_SERVER *) oldsrv = 0x000000010044c790
      (lldb) print *newsrv;
      (lcbvb_SERVER) $0 = {
        svc = {
          data = 0
          mgmt = 8091
          views = 0
          ixquery = 0
          ixadmin = 0
          n1ql = 0
          fts = 0
          cbas = 0
          views_base_ = 0x0000000000000000 <no value available>
          query_base_ = 0x0000000000000000 <no value available>
          fts_base_ = 0x0000000000000000 <no value available>
          cbas_base_ = 0x0000000000000000 <no value available>
          hoststrs = ([0] = <no value available>, [1] = <no value available>, [2] = <no value available>, [3] = <no value available>, [4] = <no value available>, [5] = <no value available>, [6] = <no value available>, [7] = <no value available>)
        }
        svc_ssl = {
          data = 0
          mgmt = 0
          views = 0
          ixquery = 0
          ixadmin = 0
          n1ql = 0
          fts = 0
          cbas = 0
          views_base_ = 0x0000000000000000 <no value available>
          query_base_ = 0x0000000000000000 <no value available>
          fts_base_ = 0x0000000000000000 <no value available>
          cbas_base_ = 0x0000000000000000 <no value available>
          hoststrs = ([0] = <no value available>, [1] = <no value available>, [2] = <no value available>, [3] = <no value available>, [4] = <no value available>, [5] = <no value available>, [6] = <no value available>, [7] = <no value available>)
        }
        authority = 0x0000000000000000 <no value available>
        hostname = 0x00000001004464f0 "10.142.171.101"
        viewpath = 0x0000000000000000 <no value available>
        querypath = 0x0000000000000000 <no value available>
        ftspath = 0x0000000000000000 <no value available>
        cbaspath = 0x0000000000000000 <no value available>
        nvbs = 0
      }
      (lldb) print *oldserv;
      error: use of undeclared identifier 'oldserv'
      (lldb) print *oldsrv;
      (lcbvb_SERVER) $1 = {
        svc = {
          data = 11210
          mgmt = 8091
          views = 8092
          ixquery = 9101
          ixadmin = 9100
          n1ql = 8093
          fts = 8094
          cbas = 0
          views_base_ = 0x0000000000000000 <no value available>
          query_base_ = 0x0000000000000000 <no value available>
          fts_base_ = 0x0000000000000000 <no value available>
          cbas_base_ = 0x0000000000000000 <no value available>
          hoststrs = ([0] = "10.142.171.101:11210", [1] = <no value available>, [2] = "10.142.171.101:8091", [3] = <no value available>, [4] = <no value available>, [5] = <no value available>, [6] = <no value available>, [7] = <no value available>)
        }
        svc_ssl = {
          data = 11207
          mgmt = 18091
          views = 18092
          ixquery = 0
          ixadmin = 0
          n1ql = 18093
          fts = 18094
          cbas = 0
          views_base_ = 0x0000000000000000 <no value available>
          query_base_ = 0x0000000000000000 <no value available>
          fts_base_ = 0x0000000000000000 <no value available>
          cbas_base_ = 0x0000000000000000 <no value available>
          hoststrs = ([0] = <no value available>, [1] = <no value available>, [2] = <no value available>, [3] = <no value available>, [4] = <no value available>, [5] = <no value available>, [6] = <no value available>, [7] = <no value available>)
        }
        authority = 0x000000010044ce70 "10.142.171.101:11210"
        hostname = 0x0000000100409a40 "10.142.171.101"
        viewpath = 0x0000000100409a70 "/default"
        querypath = 0x0000000100466df0 "/query/service"
        ftspath = 0x0000000100466e00 "/"
        cbaspath = 0x0000000000000000 <no value available>
        nvbs = 1024
      }
      

        Attachments

        For Gerrit Dashboard: CCBC-876
        # Subject Branch Project Status CR V

          Activity

          Hide
          Ellis.Breen Ellis Breen added a comment -

          More backtrace:
          (lldb) frame select 2
          frame #2: 0x00000001052f5278 libcouchbase.2.dylib`lcbvb_compare(from=0x000000010047cc90, to=0x0000000100436c30) at vbucket.c:1076
             1073	    nservers = (from->nsrv > to->nsrv ? from->nsrv : to->nsrv) + 1;
             1074	    ret->servers_added = calloc(nservers, sizeof(*ret->servers_added));
             1075	    ret->servers_removed = calloc(nservers, sizeof(*ret->servers_removed));
          -> 1076	    compute_vb_list_diff(from, to, ret->servers_added);
             1077	    compute_vb_list_diff(to, from, ret->servers_removed);
             1078
             1079	    if (to->nsrv == from->nsrv) {
          (lldb) frame variable
          (lcbvb_CONFIG *) from = 0x000000010047cc90
          (lcbvb_CONFIG *) to = 0x0000000100436c30
          (int) nservers = 2
          (lcbvb_CONFIGDIFF *) ret = 0x00000001019297b0
          (unsigned int) ii = 0
          (lldb) frame select 3
          frame #3: 0x0000000105356969 libcouchbase.2.dylib`lcb::clconfig::Confmon::do_set_next(this=0x0000000100576000, new_config=0x0000000100419470, notify_miss=false) at confmon.cc:134
             131     }
             132     if (config) {
             133         lcbvb_CHANGETYPE chstatus = LCBVB_NO_CHANGES;
          -> 134         lcbvb_CONFIGDIFF *diff = lcbvb_compare(config->vbc, new_config->vbc);
             135 
             136         if (!diff) {
             137             lcb_log(LOGARGS(this, DEBUG), "Couldn't create vbucket diff");
          (lldb) frame variable
          (lcb::clconfig::Confmon *) this = 0x0000000100576000
          (lcb::clconfig::ConfigInfo *) new_config = 0x0000000100419470
          (bool) notify_miss = false
          (unsigned int) ii = 87314368
          (lcbvb_CHANGETYPE) chstatus = LCBVB_NO_CHANGES
          (lcbvb_CONFIGDIFF *) diff = 0x000000010054adc8
          (lldb) frame select 4
          frame #4: 0x0000000105354c63 libcouchbase.2.dylib`lcb::clconfig::Confmon::do_next_provider(this=0x0000000100576000) at confmon.cc:254
             251             continue;
             252         }
             253 
          -> 254         if (do_set_next(info, false)) {
             255             LOG(this, DEBUG, "Using cached configuration");
             256         }
             257     }
          (lldb) frame variable
          (lcb::clconfig::Confmon *) this = 0x0000000100576000
          (std::__1::list<lcb::clconfig::Provider *, std::__1::allocator<lcb::clconfig::Provider *> >::const_iterator) ii = {
            __ptr_ = 0x0000000100476340
          }
          (lcb::clconfig::ConfigInfo *) info = 0x0000000100419470
          (ClusterAdminProvider *) cached_provider = 0x0000000100526760
          (lldb) frame select 5
          frame #5: 0x0000000105359047 libcouchbase.2.dylib`lcb::io::Timer<lcb::clconfig::Confmon, &(lcb::clconfig::Confmon::do_next_provider())>::cb(arg=0x0000000100576000) at timer-cxx.h:80
             77  private:
             78      static void cb(void *arg) {
             79          T *obj = reinterpret_cast<T*>(arg);
          -> 80          (obj->*M)();
             81      }
             82      Timer(const Timer&);
             83  };
          (lldb) frame variable
          (void *) arg = 0x0000000100576000
          (lcb::clconfig::Confmon *) obj = 0x0000000100576000
          (lldb)
          (void *) arg = 0x0000000100576000
          (lcb::clconfig::Confmon *) obj = 0x0000000100576000
          (lldb) frame select 11
          frame #11: 0x00000001052c4df8 _libcouchbase.cpython-36m-darwin.so`pycbc_oputil_wait_common(self=0x00000001056af750) at oputil.c:461
             458 	    */
             459 
             460     PYCBC_CONN_THR_BEGIN(self);
          -> 461     lcb_wait3(self->instance, LCB_WAIT_NOCHECK);
             462     PYCBC_CONN_THR_END(self);
             463 }
             464 
          (lldb) frame variable
          (pycbc_Bucket *) self = 0x00000001056af750
          (lldb) print *self;
          (pycbc_Bucket) $2 = {
            ob_base = {
              ob_refcnt = 8
              ob_type = 0x0000000100457ad8
            }
            instance = 0x0000000100564a30
            tc = 0x0000000000000000
            dfl_fmt = 0x00000001056c3190
            conncb = 0x0000000000000000
            dtorcb = 0x0000000000000000
            dur_testhook = 0x0000000000000000
            bucket = 0x000000010562bed8
            btype = 0x0000000100258cf0
            pipeline_queue = 0x0000000000000000
            iopswrap = 0x0000000000000000
            thrstate = 0x0000000101001a00
            lock = 0x000000010055b420
            lockmode = 1
            quiet = 0
            unlock_gil = 1
            data_passthrough = 0
            init_called = '\x01'
            nremaining = 0
            flags = 16
            dur_global = (persist_to = '\0', replicate_to = '\0')
            dur_timeout = 0
          }
          (lldb) print *(self->instance);
          (lcb_st) $3 = {
            cmdq = {
              pipelines = 0x0000000000000000
              scheds = 0x0000000000000000 <no value available>
              ctxenter = 0
              npipelines = 0
              _npipelines_ex = 0
              seq = 0
              config = 0x0000000000000000
              cqdata = 0x0000000100564a30
              fallback = 0x0000000100572a80
            }
            cookie = 0x00000001056af750
            confmon = 0x0000000100576000
            mc_nodes = 0x0000000100549680
            ht_nodes = 0x0000000100556a40
            cur_configinfo = 0x0000000000000000
            bs_state = 0x00000001005656a0
            callbacks = {
              v3callbacks = {
                [0] = 0x0000000105320100 (libcouchbase.2.dylib`compat_default_callback at callbacks.c:88)
                [1] = 0x00000001052b71e0 (_libcouchbase.cpython-36m-darwin.so`value_callback at callbacks.c:384)
                [2] = 0x00000001052b7050 (_libcouchbase.cpython-36m-darwin.so`durability_chain_common at callbacks.c:361)
                [3] = 0x00000001052b71e0 (_libcouchbase.cpython-36m-darwin.so`value_callback at callbacks.c:384)
                [4] = 0x00000001052b7100 (_libcouchbase.cpython-36m-darwin.so`keyop_simple_callback at callbacks.c:532)
                [5] = 0x00000001052b7050 (_libcouchbase.cpython-36m-darwin.so`durability_chain_common at callbacks.c:361)
                [6] = 0x00000001052b7100 (_libcouchbase.cpython-36m-darwin.so`keyop_simple_callback at callbacks.c:532)
                [7] = 0x00000001052b7570 (_libcouchbase.cpython-36m-darwin.so`stats_callback at callbacks.c:561)
                [8] = 0x0000000000000000
                [9] = 0x0000000000000000
                [10] = 0x0000000000000000
                [11] = 0x00000001052b73e0 (_libcouchbase.cpython-36m-darwin.so`observe_callback at callbacks.c:626)
                [12] = 0x00000001052b71e0 (_libcouchbase.cpython-36m-darwin.so`value_callback at callbacks.c:384)
                [13] = 0x00000001052b7100 (_libcouchbase.cpython-36m-darwin.so`keyop_simple_callback at callbacks.c:532)
                [14] = 0x00000001052ca330 (_libcouchbase.cpython-36m-darwin.so`complete_callback at http.c:167)
                [15] = 0x0000000000000000
                [16] = 0x0000000000000000
                [17] = 0x0000000000000000
                [18] = 0x00000001052b7bb0 (_libcouchbase.cpython-36m-darwin.so`subdoc_callback at callbacks.c:471)
                [19] = 0x00000001052b7bb0 (_libcouchbase.cpython-36m-darwin.so`subdoc_callback at callbacks.c:471)
                [20] = 0x0000000000000000
                [21] = 0x00000001052b7810 (_libcouchbase.cpython-36m-darwin.so`ping_callback at callbacks.c:733)
              }
              get = 0x000000010531fee0 (libcouchbase.2.dylib`dummy_get_callback at callbacks.c:60)
              store = 0x000000010531ff00 (libcouchbase.2.dylib`dummy_store_callback at callbacks.c:31)
              arithmetic = 0x000000010531ff20 (libcouchbase.2.dylib`dummy_arithmetic_callback at callbacks.c:61)
              observe = 0x0000000105320060 (libcouchbase.2.dylib`dummy_observe_callback at callbacks.c:66)
              remove = 0x000000010531ff40 (libcouchbase.2.dylib`dummy_remove_callback at callbacks.c:62)
              stat = 0x000000010531ffb0 (libcouchbase.2.dylib`dummy_stat_callback at callbacks.c:57)
              version = 0x000000010531ffd0 (libcouchbase.2.dylib`dummy_version_callback at callbacks.c:58)
              touch = 0x000000010531ff60 (libcouchbase.2.dylib`dummy_touch_callback at callbacks.c:63)
              flush = 0x0000000105320010 (libcouchbase.2.dylib`dummy_flush_callback at callbacks.c:64)
              error = 0x000000010531ff80 (libcouchbase.2.dylib`dummy_error_callback at callbacks.c:24)
              http_complete = 0x000000010531fff0 (libcouchbase.2.dylib`dummy_http_callback at callbacks.c:37)
              http_data = 0x000000010531fff0 (libcouchbase.2.dylib`dummy_http_callback at callbacks.c:37)
              unlock = 0x0000000105320030 (libcouchbase.2.dylib`dummy_unlock_callback at callbacks.c:65)
              configuration = 0x0000000105320050 (libcouchbase.2.dylib`dummy_configuration_callback at callbacks.c:41)
              verbosity = 0x0000000105320080 (libcouchbase.2.dylib`dummy_verbosity_callback at callbacks.c:59)
              durability = 0x00000001053200a0 (libcouchbase.2.dylib`dummy_durability_callback at callbacks.c:67)
              errmap = 0x0000000105376c10 (libcouchbase.2.dylib`::lcb_errmap_default(lcb_t, lcb_uint16_t) at handler.cc:92)
              bootstrap = 0x00000001052b7df0 (_libcouchbase.cpython-36m-darwin.so`bootstrap_callback at callbacks.c:698)
              pktfwd = 0x00000001053200e0 (libcouchbase.2.dylib`dummy_pktfwd_callback at callbacks.c:50)
              pktflushed = 0x00000001053200d0 (libcouchbase.2.dylib`dummy_pktflushed_callback at callbacks.c:53)
            }
            kv_timings = 0x0000000000000000
            pendops = {
              items = ([0] = 0x0000000100554c80, [1] = 0x00000001005695b0, [2] = 0x000000010054b040, [3] = 0x0000000100576600)
              count = 1
            }
            wait = 1
            memd_sockpool = 0x000000010052b790
            http_sockpool = 0x000000010056c820
            last_error = LCB_SUCCESS
            settings = 0x000000010057af50
            iotable = 0x0000000100568800
            retryq = 0x000000010051f430
            scratch = 0x0000000100441090
            vbguess = 0x0000000000000000
            n1ql_cache = 0x000000010052abb0
            dcpinfo = 0x0000000000000000
            dtor_timer = 0x0000000000000000
            type = 1
            btype = LCB_BTYPE_UNSPEC
          }
          

          Show
          Ellis.Breen Ellis Breen added a comment - More backtrace: (lldb) frame select 2 frame # 2 : 0x00000001052f5278 libcouchbase. 2 .dylib`lcbvb_compare(from= 0x000000010047cc90 , to= 0x0000000100436c30 ) at vbucket.c: 1076    1073     nservers = (from->nsrv > to->nsrv ? from->nsrv : to->nsrv) + 1 ;    1074     ret->servers_added = calloc(nservers, sizeof(*ret->servers_added));    1075     ret->servers_removed = calloc(nservers, sizeof(*ret->servers_removed)); -> 1076     compute_vb_list_diff(from, to, ret->servers_added);    1077     compute_vb_list_diff(to, from, ret->servers_removed);    1078    1079     if (to->nsrv == from->nsrv) { (lldb) frame variable (lcbvb_CONFIG *) from = 0x000000010047cc90 (lcbvb_CONFIG *) to = 0x0000000100436c30 ( int ) nservers = 2 (lcbvb_CONFIGDIFF *) ret = 0x00000001019297b0 (unsigned int ) ii = 0 (lldb) frame select 3 frame # 3 : 0x0000000105356969 libcouchbase. 2 .dylib`lcb::clconfig::Confmon::do_set_next( this = 0x0000000100576000 , new_config= 0x0000000100419470 , notify_miss= false ) at confmon.cc: 134    131     }    132     if (config) {    133         lcbvb_CHANGETYPE chstatus = LCBVB_NO_CHANGES; -> 134         lcbvb_CONFIGDIFF *diff = lcbvb_compare(config->vbc, new_config->vbc);    135      136         if (!diff) {    137             lcb_log(LOGARGS( this , DEBUG), "Couldn't create vbucket diff" ); (lldb) frame variable (lcb::clconfig::Confmon *) this = 0x0000000100576000 (lcb::clconfig::ConfigInfo *) new_config = 0x0000000100419470 (bool) notify_miss = false (unsigned int ) ii = 87314368 (lcbvb_CHANGETYPE) chstatus = LCBVB_NO_CHANGES (lcbvb_CONFIGDIFF *) diff = 0x000000010054adc8 (lldb) frame select 4 frame # 4 : 0x0000000105354c63 libcouchbase. 2 .dylib`lcb::clconfig::Confmon::do_next_provider( this = 0x0000000100576000 ) at confmon.cc: 254    251             continue ;    252         }    253   -> 254         if (do_set_next(info, false )) {    255             LOG( this , DEBUG, "Using cached configuration" );    256         }    257     } (lldb) frame variable (lcb::clconfig::Confmon *) this = 0x0000000100576000 (std::__1::list<lcb::clconfig::Provider *, std::__1::allocator<lcb::clconfig::Provider *> >::const_iterator) ii = {   __ptr_ = 0x0000000100476340 } (lcb::clconfig::ConfigInfo *) info = 0x0000000100419470 (ClusterAdminProvider *) cached_provider = 0x0000000100526760 (lldb) frame select 5 frame # 5 : 0x0000000105359047 libcouchbase. 2 .dylib`lcb::io::Timer<lcb::clconfig::Confmon, &(lcb::clconfig::Confmon::do_next_provider())>::cb(arg= 0x0000000100576000 ) at timer-cxx.h: 80    77   private :    78       static void cb( void *arg) {    79           T *obj = reinterpret_cast<T*>(arg); -> 80           (obj->*M)();    81       }    82       Timer( const Timer&);    83   }; (lldb) frame variable ( void *) arg = 0x0000000100576000 (lcb::clconfig::Confmon *) obj = 0x0000000100576000 (lldb) ( void *) arg = 0x0000000100576000 (lcb::clconfig::Confmon *) obj = 0x0000000100576000 (lldb) frame select 11 frame # 11 : 0x00000001052c4df8 _libcouchbase.cpython-36m-darwin.so`pycbc_oputil_wait_common(self= 0x00000001056af750 ) at oputil.c: 461    458       */    459      460     PYCBC_CONN_THR_BEGIN(self); -> 461     lcb_wait3(self->instance, LCB_WAIT_NOCHECK);    462     PYCBC_CONN_THR_END(self);    463 }    464   (lldb) frame variable (pycbc_Bucket *) self = 0x00000001056af750 (lldb) print *self; (pycbc_Bucket) $ 2 = {   ob_base = {     ob_refcnt = 8     ob_type = 0x0000000100457ad8   }   instance = 0x0000000100564a30   tc = 0x0000000000000000   dfl_fmt = 0x00000001056c3190   conncb = 0x0000000000000000   dtorcb = 0x0000000000000000   dur_testhook = 0x0000000000000000   bucket = 0x000000010562bed8   btype = 0x0000000100258cf0   pipeline_queue = 0x0000000000000000   iopswrap = 0x0000000000000000   thrstate = 0x0000000101001a00   lock = 0x000000010055b420   lockmode = 1   quiet = 0   unlock_gil = 1   data_passthrough = 0   init_called = '\x01'   nremaining = 0   flags = 16   dur_global = (persist_to = '\0' , replicate_to = '\0' )   dur_timeout = 0 } (lldb) print *(self->instance); (lcb_st) $ 3 = {   cmdq = {     pipelines = 0x0000000000000000     scheds = 0x0000000000000000 <no value available>     ctxenter = 0     npipelines = 0     _npipelines_ex = 0     seq = 0     config = 0x0000000000000000     cqdata = 0x0000000100564a30     fallback = 0x0000000100572a80   }   cookie = 0x00000001056af750   confmon = 0x0000000100576000   mc_nodes = 0x0000000100549680   ht_nodes = 0x0000000100556a40   cur_configinfo = 0x0000000000000000   bs_state = 0x00000001005656a0   callbacks = {     v3callbacks = {       [ 0 ] = 0x0000000105320100 (libcouchbase. 2 .dylib`compat_default_callback at callbacks.c: 88 )       [ 1 ] = 0x00000001052b71e0 (_libcouchbase.cpython-36m-darwin.so`value_callback at callbacks.c: 384 )       [ 2 ] = 0x00000001052b7050 (_libcouchbase.cpython-36m-darwin.so`durability_chain_common at callbacks.c: 361 )       [ 3 ] = 0x00000001052b71e0 (_libcouchbase.cpython-36m-darwin.so`value_callback at callbacks.c: 384 )       [ 4 ] = 0x00000001052b7100 (_libcouchbase.cpython-36m-darwin.so`keyop_simple_callback at callbacks.c: 532 )       [ 5 ] = 0x00000001052b7050 (_libcouchbase.cpython-36m-darwin.so`durability_chain_common at callbacks.c: 361 )       [ 6 ] = 0x00000001052b7100 (_libcouchbase.cpython-36m-darwin.so`keyop_simple_callback at callbacks.c: 532 )       [ 7 ] = 0x00000001052b7570 (_libcouchbase.cpython-36m-darwin.so`stats_callback at callbacks.c: 561 )       [ 8 ] = 0x0000000000000000       [ 9 ] = 0x0000000000000000       [ 10 ] = 0x0000000000000000       [ 11 ] = 0x00000001052b73e0 (_libcouchbase.cpython-36m-darwin.so`observe_callback at callbacks.c: 626 )       [ 12 ] = 0x00000001052b71e0 (_libcouchbase.cpython-36m-darwin.so`value_callback at callbacks.c: 384 )       [ 13 ] = 0x00000001052b7100 (_libcouchbase.cpython-36m-darwin.so`keyop_simple_callback at callbacks.c: 532 )       [ 14 ] = 0x00000001052ca330 (_libcouchbase.cpython-36m-darwin.so`complete_callback at http.c: 167 )       [ 15 ] = 0x0000000000000000       [ 16 ] = 0x0000000000000000       [ 17 ] = 0x0000000000000000       [ 18 ] = 0x00000001052b7bb0 (_libcouchbase.cpython-36m-darwin.so`subdoc_callback at callbacks.c: 471 )       [ 19 ] = 0x00000001052b7bb0 (_libcouchbase.cpython-36m-darwin.so`subdoc_callback at callbacks.c: 471 )       [ 20 ] = 0x0000000000000000       [ 21 ] = 0x00000001052b7810 (_libcouchbase.cpython-36m-darwin.so`ping_callback at callbacks.c: 733 )     }     get = 0x000000010531fee0 (libcouchbase. 2 .dylib`dummy_get_callback at callbacks.c: 60 )     store = 0x000000010531ff00 (libcouchbase. 2 .dylib`dummy_store_callback at callbacks.c: 31 )     arithmetic = 0x000000010531ff20 (libcouchbase. 2 .dylib`dummy_arithmetic_callback at callbacks.c: 61 )     observe = 0x0000000105320060 (libcouchbase. 2 .dylib`dummy_observe_callback at callbacks.c: 66 )     remove = 0x000000010531ff40 (libcouchbase. 2 .dylib`dummy_remove_callback at callbacks.c: 62 )     stat = 0x000000010531ffb0 (libcouchbase. 2 .dylib`dummy_stat_callback at callbacks.c: 57 )     version = 0x000000010531ffd0 (libcouchbase. 2 .dylib`dummy_version_callback at callbacks.c: 58 )     touch = 0x000000010531ff60 (libcouchbase. 2 .dylib`dummy_touch_callback at callbacks.c: 63 )     flush = 0x0000000105320010 (libcouchbase. 2 .dylib`dummy_flush_callback at callbacks.c: 64 )     error = 0x000000010531ff80 (libcouchbase. 2 .dylib`dummy_error_callback at callbacks.c: 24 )     http_complete = 0x000000010531fff0 (libcouchbase. 2 .dylib`dummy_http_callback at callbacks.c: 37 )     http_data = 0x000000010531fff0 (libcouchbase. 2 .dylib`dummy_http_callback at callbacks.c: 37 )     unlock = 0x0000000105320030 (libcouchbase. 2 .dylib`dummy_unlock_callback at callbacks.c: 65 )     configuration = 0x0000000105320050 (libcouchbase. 2 .dylib`dummy_configuration_callback at callbacks.c: 41 )     verbosity = 0x0000000105320080 (libcouchbase. 2 .dylib`dummy_verbosity_callback at callbacks.c: 59 )     durability = 0x00000001053200a0 (libcouchbase. 2 .dylib`dummy_durability_callback at callbacks.c: 67 )     errmap = 0x0000000105376c10 (libcouchbase. 2 .dylib`::lcb_errmap_default(lcb_t, lcb_uint16_t) at handler.cc: 92 )     bootstrap = 0x00000001052b7df0 (_libcouchbase.cpython-36m-darwin.so`bootstrap_callback at callbacks.c: 698 )     pktfwd = 0x00000001053200e0 (libcouchbase. 2 .dylib`dummy_pktfwd_callback at callbacks.c: 50 )     pktflushed = 0x00000001053200d0 (libcouchbase. 2 .dylib`dummy_pktflushed_callback at callbacks.c: 53 )   }   kv_timings = 0x0000000000000000   pendops = {     items = ([ 0 ] = 0x0000000100554c80 , [ 1 ] = 0x00000001005695b0 , [ 2 ] = 0x000000010054b040 , [ 3 ] = 0x0000000100576600 )     count = 1   }   wait = 1   memd_sockpool = 0x000000010052b790   http_sockpool = 0x000000010056c820   last_error = LCB_SUCCESS   settings = 0x000000010057af50   iotable = 0x0000000100568800   retryq = 0x000000010051f430   scratch = 0x0000000100441090   vbguess = 0x0000000000000000   n1ql_cache = 0x000000010052abb0   dcpinfo = 0x0000000000000000   dtor_timer = 0x0000000000000000   type = 1   btype = LCB_BTYPE_UNSPEC }
          Hide
          avsej Sergey Avseyev added a comment -

          Ellis Breen, could you attach configuration, which caused the issue?

          Show
          avsej Sergey Avseyev added a comment - Ellis Breen , could you attach configuration, which caused the issue?
          Hide
          Ellis.Breen Ellis Breen added a comment -

          Looks like a call to bucket_remove on a non-existent bucket (default).

          test_create_ephemeral_bucket_and_use (couchbase.tests.test_sync.transplant_class.<locals>.C) ... test_create_ephemeral_bucket_and_use (couchbase.tests.test_sync.transplant_class.<locals>.C) ... 522ms [I13] {10307/307} [INFO] (instance - L:423) Version=2.8.3, Changeset=8ce52f9294cfef59b9ff44f3cfc18cc3ca8c2860
          522ms [I13] {10307/307} [INFO] (instance - L:424) Effective connection string: http://10.142.171.101:8091/default?ipv6=allow. Bucket=default
          573ms [I13] {10307/307} [INFO] (instance - L:137) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records
          573ms [I13] {10307/307} [DEBUG] (instance - L:77) Adding host 10.142.171.101:8091 to initial HTTP bootstrap list
          573ms [I13] {10307/307} [DEBUG] (instance - L:77) Adding host 10.142.171.101:11210 to initial CCCP bootstrap list
          573ms [I13] {10307/307} [TRACE] (instance - L:117) Bootstrap hosts loaded (cccp:1, http:1)
           
          
          

          CCBC-876.pcapng

          Show
          Ellis.Breen Ellis Breen added a comment - Looks like a call to bucket_remove on a non-existent bucket (default). test_create_ephemeral_bucket_and_use (couchbase.tests.test_sync.transplant_class.<locals>.C) ... test_create_ephemeral_bucket_and_use (couchbase.tests.test_sync.transplant_class.<locals>.C) ... 522ms [I13] { 10307 / 307 } [INFO] (instance - L: 423 ) Version= 2.8 . 3 , Changeset=8ce52f9294cfef59b9ff44f3cfc18cc3ca8c2860 522ms [I13] { 10307 / 307 } [INFO] (instance - L: 424 ) Effective connection string: http: //10.142.171.101:8091/default?ipv6=allow. Bucket=default 573ms [I13] { 10307 / 307 } [INFO] (instance - L: 137 ) DNS SRV lookup failed: DNS/Hostname lookup failed. Ignore this if not relying on DNS SRV records 573ms [I13] { 10307 / 307 } [DEBUG] (instance - L: 77 ) Adding host 10.142 . 171.101 : 8091 to initial HTTP bootstrap list 573ms [I13] { 10307 / 307 } [DEBUG] (instance - L: 77 ) Adding host 10.142 . 171.101 : 11210 to initial CCCP bootstrap list 573ms [I13] { 10307 / 307 } [TRACE] (instance - L: 117 ) Bootstrap hosts loaded (cccp: 1 , http: 1 )   CCBC-876.pcapng
          Hide
          Ellis.Breen Ellis Breen added a comment - - edited

          Using stock 4 node config from couchbaselabs/vagrants/ubuntu14, LCB 2.8.3 on MacOS Sierra

          Show
          Ellis.Breen Ellis Breen added a comment - - edited Using stock 4 node config from couchbaselabs/vagrants/ubuntu14, LCB 2.8.3 on MacOS Sierra

            People

            • Assignee:
              avsej Sergey Avseyev
              Reporter:
              Ellis.Breen Ellis Breen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty

                  Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.