Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37197

[Sytem Test]: runtime error: slice bounds out of range stack trace

    XMLWordPrintable

Details

    Description

      Build: 6.5.0-4926 not seen on 6.5.0-4908

      Test: MH longevity

      Day: 2nd

      Cycle: 2nd

      observe panic on 172.23.98.135

      [worker_timer_op_0:/tmp/127.0.0.1:8091_0_355489654.sock:21786] readMessageLoop recover, runtime error: slice bounds out of range stack trace: goroutine 2966991 [running]:runtime/debug.Stack(0xc42f3bfd28, 0x12a24e0, 0x1d11270) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/debug/stack.go:24 +0xa7github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop.func1(0x140f0b1, 0x19, 0xc433aea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:784 +0x6epanic(0x12a24e0, 0x1d11270) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/panic.go:502 +0x229github.com/google/flatbuffers/go.(*Table).GetSOffsetT(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/google/flatbuffers/go/table.go:139github.com/google/flatbuffers/go.(*Table).Offset(0xc42f3bfe88, 0x4, 0xdc) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/google/flatbuffers/go/table.go:15 +0x134github.com/couchbase/eventing/gen/flatbuf/response.(*Response).MsgType(0xc42f3bfe88, 0xa26fac) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/gen/flatbuf/response/Response.go:30 +0x38github.com/couchbase/eventing/consumer.(*Consumer).parseWorkerResponse(0xc433aea000, 0xc432591004, 0x10, 0xffc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/protocol.go:383 +0x8agithub.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop(0xc433aea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:820 +0x1c7created by github.com/couchbase/eventing/consumer.(*Consumer).SetConnHandle /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/exported_functions.go:282 +0x3322019-12-07T02:41:19.048-08:00 [Error] Consumer::readMessageLoop [worker_timer_op_0:/tmp/127.0.0.1:8091_0_355489654.sock:21786] readMessageLoop recover, runtime error: slice bounds out of range stack trace: goroutine 2967221 [running]:runtime/debug.Stack(0xc42af77d28, 0x12a24e0, 0x1d11270) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/debug/stack.go:24 +0xa7github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop.func1(0x140f0b1, 0x19, 0xc433aea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:784 +0x6epanic(0x12a24e0, 0x1d11270) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/panic.go:502 +0x229github.com/google/flatbuffers/go.(*Table).GetVOffsetT(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/google/flatbuffers/go/table.go:134github.com/google/flatbuffers/go.(*Table).Offset(0xc42af77e88, 0x4, 0xdc) /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/google/flatbuffers/go/table.go:16 +0x111github.com/couchbase/eventing/gen/flatbuf/response.(*Response).MsgType(0xc42af77e88, 0xa26fac) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/gen/flatbuf/response/Response.go:30 +0x38github.com/couchbase/eventing/consumer.(*Consumer).parseWorkerResponse(0xc433aea000, 0xc425ef0004, 0x28, 0x7b4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/protocol.go:383 +0x8agithub.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop(0xc433aea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:820 +0x1c7created by github.com/couchbase/eventing/consumer.(*Consumer).SetConnHandle /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/exported_functions.go:282 +0x3322019-12-07T02:41:19.049-08:00 [Info] Consumer::processReqStreamMessages [worker_bucket_op_curl_0:/tmp/127.0.0.1:8091_0_42136478.sock:12547] vb: 768 reqStreamCh size: 28 msg:  

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            We saw the following stack trace -

            #0  0x00007f943deea5f7 in raise () from /usr/lib64/libc-2.17.so
            #1  0x00007f943deebce8 in abort () from /usr/lib64/libc-2.17.so
            #2  0x00007f943df2a317 in __libc_message () from /usr/lib64/libc-2.17.so
            #3  0x00007f943df30184 in malloc_printerr () from /usr/lib64/libc-2.17.so
            #4  0x00007f943df338f6 in _int_malloc () from /usr/lib64/libc-2.17.so
            #5  0x00007f943df3487c in malloc () from /usr/lib64/libc-2.17.so
            #6  0x00007f943ea3a6a8 in operator new (sz=sz@entry=96) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/new_op.cc:50
            #7  0x0000000000450873 in allocate (this=0x7f9432ff7810, __n=4) at /usr/local/include/c++/7.3.0/ext/new_allocator.h:111
            #8  allocate (__a=..., __n=4) at /usr/local/include/c++/7.3.0/bits/alloc_traits.h:436
            #9  _M_allocate (this=0x7f9432ff7810, __n=4) at /usr/local/include/c++/7.3.0/bits/stl_vector.h:172
            #10 _M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > > const*, std::vector<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > > > > > > (this=0x7f9432ff7810, __last=..., __first=..., __n=4) at /usr/local/include/c++/7.3.0/bits/stl_vector.h:1260
            #11 std::vector<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >::operator= (this=this@entry=0x7f9432ff7810, __x=...)
                at /usr/local/include/c++/7.3.0/bits/vector.tcc:206
            #12 0x000000000049b9cf in _M_main_dispatch (
                __match_mode=std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Exact, this=0x7f9432ff7810)      at /usr/local/include/c++/7.3.0/bits/regex_executor.tcc:86
            #13 _M_main (
                __match_mode=std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Exact, this=0x7f9432ff7810)      at /usr/local/include/c++/7.3.0/bits/regex_executor.h:149
            #14 _M_match (this=0x7f9432ff7810) at /usr/local/include/c++/7.3.0/bits/regex_executor.h:93
            #15 std::__detail::__regex_algo_impl<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, char, std::__cxx11::regex_traits<char>, (std::__detail::_RegexExecutorPolicy)0, true> (__s=..., __e=..., __m=..., __re=..., __flags=__flags@entry=0) at /usr/local/include/c++/7.3.0/bits/regex.tcc:80
            #16 0x000000000049aacf in regex_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > > >, char, std::__cxx11::regex_traits<char> > (__flags=0, __re=..., __m=..., __e=..., __s=...) at /usr/local/include/c++/7.3.0/bits/regex.h:1997
            #17 regex_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> >, char, std::__cxx11::regex_traits<char> > (__flags=0, __re=..., __last=..., __first=...)
                at /usr/local/include/c++/7.3.0/bits/regex.h:2022
            #18 regex_match<std::char_traits<char>, std::allocator<char>, char, std::__cxx11::regex_traits<char> > (__flags=0, __re=..., __s=...) at /usr/local/include/c++/7.3.0/bits/regex.h:2127
            #19 JoinHostPort (host=..., port=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/src/utils.cc:288
            #20 0x000000000046e313 in GetUsername (cookie=0x7f943419d670, host=0x7f940c2f2c78 <Address 0x7f940c2f2c78 out of bounds>, port=0x7f940c2f307a <Address 0x7f940c2f307a out of bounds>,
                bucket=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/src/lcb_util.cc:24
            #21 0x00007f943f5bb60c in lcb::Authenticator::username_for[abi:cxx11](char const*, char const*, char const*, bool) (this=this@entry=0x7f943432f8b0,
                host=host@entry=0x7f940c2f2c78 <Address 0x7f940c2f2c78 out of bounds>, port=port@entry=0x7f940c2f307a <Address 0x7f940c2f307a out of bounds>, bucket=<optimized out>, use_cache=use_cache@entry=false)     at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/auth.cc:90
            #22 0x00007f943f60569d in lcb::SessionRequestImpl::setup (this=this@entry=0x7f940c179440, nistrs=..., host=..., auth=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/negotiate.cc:241
            #23 0x00007f943f6079fc in lcb::SessionRequestImpl::start (this=this@entry=0x7f940c179440, sock=sock@entry=0x7f940c04a1e0)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/negotiate.cc:708
            #24 0x00007f943f6080a2 in lcb::SessionRequest::start (sock=sock@entry=0x7f940c04a1e0, settings=0x7f943432d0f0, tmo=2500000,
                callback=callback@entry=0x7f943f603930 <on_connected(lcbio_SOCKET*, void*, lcb_error_t, lcbio_OSERR)>, data=data@entry=0x7f943434f9c0)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/negotiate.cc:744
            #25 0x00007f943f603b20 in handle_connected (syserr=<optimized out>, err=<optimized out>, sock=0x7f940c04a1e0, this=0x7f943434f9c0)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:794
            #26 on_connected (sock=0x7f940c04a1e0, data=0x7f943434f9c0, err=<optimized out>, syserr=<optimized out>)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:766
            ---Type <return> to continue, or q <return> to quit---
            #27 0x00007f943f5a59bc in invoke (this=0x7f940c06fd50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:281
            #28 connection_available (this=0x7f940c06ffe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:300
            #29 on_connected (err=<optimized out>, sock_=<optimized out>, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:340
            #30 on_connected (sock=<optimized out>, arg=<optimized out>, err=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:310
            #31 0x00007f943f5a21f9 in handler (this=0x7f940c25a5b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/connect.cc:162
            #32 lcb::io::Timer<lcb::io::Connstart, &lcb::io::Connstart::handler>::cb (arg=0x7f940c25a5b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/timer-cxx.h:80
            #33 0x00007f943f59ecad in timer_callback (sock=<optimized out>, which=<optimized out>, arg=0x7f940c28b830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/timer.c:45    #34 0x00007f943f5828f7 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/plugins/io/select/plugin-select.c:289
            #35 0x00007f943f61147e in lcb_wait (instance=0x7f943432ce40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/wait.cc:103
            #36 0x0000000000463596 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>,
                isRetriable=<optimized out>, initial_delay_milliseconds=200, max_retry_count=5)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40
            #37 0x000000000046581d in timer::TimerStore::Upsert (this=this@entry=0x7f9434322b30, key=..., value=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:362
            #38 0x000000000046683a in timer::TimerStore::SetTimer (this=0x7f9434322b30, timer=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:72
            #39 0x000000000041f846 in V8Worker::SetTimer (this=this@entry=0x7f94340b09a0, tinfo=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:1148
            #40 0x000000000043dbdf in Timer::CreateTimerImpl (this=0x7f9434223020, args=...)
                at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:98
            #41 0x000000000043e286 in CreateTimer (args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:142
            #42 0x00007f9440d65239 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) () from /opt/couchbase/lib/libv8.so
            #43 0x00007f9440d64738 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
               from /opt/couchbase/lib/libv8.so
            #44 0x00007f9440d63ec6 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) () from /opt/couchbase/lib/libv8.so
            #45 0x00007f944159d8ae in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit () from /opt/couchbase/lib/libv8.so
            #46 0x00002ce55018816e in ?? ()
            #47 0x00002829aa1025a1 in ?? ()
            #48 0x0000106b2a31d139 in ?? ()
            #49 0x0000000900000000 in ?? ()
            #50 0x00002829aa102681 in ?? ()
            #51 0x00002fe24f4fee09 in ?? ()
            #52 0x0000195cae456e91 in ?? ()
            #53 0x00002fe24f4fed11 in ?? ()
            #54 0x00002e9736e04ad1 in ?? ()
            #55 0x00001e19ba08b1e1 in ?? ()
            #56 0x00002fe24f4fee09 in ?? ()
            #57 0x0000195cae456e91 in ?? ()
            #58 0x00002fe24f4fed11 in ?? ()
            #59 0x00002e9736e04ad1 in ?? ()
            #60 0x0000106b2a31d139 in ?? ()
            #61 0x00002fe24f4fee09 in ?? ()
            #62 0x0000195cae456eb9 in ?? ()
            #63 0x00002fe24f4fed11 in ?? ()
            #64 0x000000cf00000000 in ?? ()
            #65 0x00002e9736e04f59 in ?? ()
            #66 0x00002e9736e04a69 in ?? ()
            #67 0x0000106b2a302ad9 in ?? ()
            #68 0x00007f9432ff9578 in ?? ()
            ---Type <return> to continue, or q <return> to quit---
            #69 0x00007f944150d603 in Builtins_JSEntryTrampoline () from /opt/couchbase/lib/libv8.so
            #70 0x00002fe24f4feca1 in ?? ()
            #71 0x00001e19ba08b1e1 in ?? ()
            #72 0x00002e9736e04a69 in ?? ()
            #73 0x0000000000000020 in ?? ()
            #74 0x00007f9432ff95e0 in ?? ()
            #75 0x00002ce5501840de in ?? ()
            #76 0x0000000000000000 in ?? ()
            

            We see the first occurrence of Address 0x7f940c2f2c78 out of bounds at frame 21 -

            #21 0x00007f943f5bb60c in lcb::Authenticator::username_for[abi:cxx11](char const*, char const*, char const*, bool) (this=this@entry=0x7f943432f8b0,
                host=host@entry=0x7f940c2f2c78 <Address 0x7f940c2f2c78 out of bounds>, port=port@entry=0x7f940c2f307a <Address 0x7f940c2f307a out of bounds>, bucket=<optimized out>, use_cache=use_cache@entry=false)     at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/auth.cc:90
            

            I've attached the core dump for eventing-consumer - eventing-consumer-CB-4926.core
            Requesting SDK team to have a look and to assign back to us if the problem lies with eventing.

            Gautham.Banasandra Gautham Banasandra (Inactive) added a comment - - edited We saw the following stack trace - #0 0x00007f943deea5f7 in raise () from /usr/lib64/libc-2.17.so #1 0x00007f943deebce8 in abort () from /usr/lib64/libc-2.17.so #2 0x00007f943df2a317 in __libc_message () from /usr/lib64/libc-2.17.so #3 0x00007f943df30184 in malloc_printerr () from /usr/lib64/libc-2.17.so #4 0x00007f943df338f6 in _int_malloc () from /usr/lib64/libc-2.17.so #5 0x00007f943df3487c in malloc () from /usr/lib64/libc-2.17.so #6 0x00007f943ea3a6a8 in operator new (sz=sz@entry=96) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/libsupc++/new_op.cc:50 #7 0x0000000000450873 in allocate (this=0x7f9432ff7810, __n=4) at /usr/local/include/c++/7.3.0/ext/new_allocator.h:111 #8 allocate (__a=..., __n=4) at /usr/local/include/c++/7.3.0/bits/alloc_traits.h:436 #9 _M_allocate (this=0x7f9432ff7810, __n=4) at /usr/local/include/c++/7.3.0/bits/stl_vector.h:172 #10 _M_allocate_and_copy<__gnu_cxx::__normal_iterator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > > const*, std::vector<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > > > > > > (this=0x7f9432ff7810, __last=..., __first=..., __n=4) at /usr/local/include/c++/7.3.0/bits/stl_vector.h:1260 #11 std::vector<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >::operator= (this=this@entry=0x7f9432ff7810, __x=...) at /usr/local/include/c++/7.3.0/bits/vector.tcc:206 #12 0x000000000049b9cf in _M_main_dispatch ( __match_mode=std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Exact, this=0x7f9432ff7810) at /usr/local/include/c++/7.3.0/bits/regex_executor.tcc:86 #13 _M_main ( __match_mode=std::__detail::_Executor<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::__cxx11::regex_traits<char>, true>::_Exact, this=0x7f9432ff7810) at /usr/local/include/c++/7.3.0/bits/regex_executor.h:149 #14 _M_match (this=0x7f9432ff7810) at /usr/local/include/c++/7.3.0/bits/regex_executor.h:93 #15 std::__detail::__regex_algo_impl<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, char, std::__cxx11::regex_traits<char>, (std::__detail::_RegexExecutorPolicy)0, true> (__s=..., __e=..., __m=..., __re=..., __flags=__flags@entry=0) at /usr/local/include/c++/7.3.0/bits/regex.tcc:80 #16 0x000000000049aacf in regex_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> >, std::allocator<std::__cxx11::sub_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> > > >, char, std::__cxx11::regex_traits<char> > (__flags=0, __re=..., __m=..., __e=..., __s=...) at /usr/local/include/c++/7.3.0/bits/regex.h:1997 #17 regex_match<__gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char> >, char, std::__cxx11::regex_traits<char> > (__flags=0, __re=..., __last=..., __first=...) at /usr/local/include/c++/7.3.0/bits/regex.h:2022 #18 regex_match<std::char_traits<char>, std::allocator<char>, char, std::__cxx11::regex_traits<char> > (__flags=0, __re=..., __s=...) at /usr/local/include/c++/7.3.0/bits/regex.h:2127 #19 JoinHostPort (host=..., port=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/src/utils.cc:288 #20 0x000000000046e313 in GetUsername (cookie=0x7f943419d670, host=0x7f940c2f2c78 <Address 0x7f940c2f2c78 out of bounds>, port=0x7f940c2f307a <Address 0x7f940c2f307a out of bounds>, bucket=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/src/lcb_util.cc:24 #21 0x00007f943f5bb60c in lcb::Authenticator::username_for[abi:cxx11](char const*, char const*, char const*, bool) (this=this@entry=0x7f943432f8b0, host=host@entry=0x7f940c2f2c78 <Address 0x7f940c2f2c78 out of bounds>, port=port@entry=0x7f940c2f307a <Address 0x7f940c2f307a out of bounds>, bucket=<optimized out>, use_cache=use_cache@entry=false) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/auth.cc:90 #22 0x00007f943f60569d in lcb::SessionRequestImpl::setup (this=this@entry=0x7f940c179440, nistrs=..., host=..., auth=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/negotiate.cc:241 #23 0x00007f943f6079fc in lcb::SessionRequestImpl::start (this=this@entry=0x7f940c179440, sock=sock@entry=0x7f940c04a1e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/negotiate.cc:708 #24 0x00007f943f6080a2 in lcb::SessionRequest::start (sock=sock@entry=0x7f940c04a1e0, settings=0x7f943432d0f0, tmo=2500000, callback=callback@entry=0x7f943f603930 <on_connected(lcbio_SOCKET*, void*, lcb_error_t, lcbio_OSERR)>, data=data@entry=0x7f943434f9c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/negotiate.cc:744 #25 0x00007f943f603b20 in handle_connected (syserr=<optimized out>, err=<optimized out>, sock=0x7f940c04a1e0, this=0x7f943434f9c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:794 #26 on_connected (sock=0x7f940c04a1e0, data=0x7f943434f9c0, err=<optimized out>, syserr=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/mcserver/mcserver.cc:766 ---Type <return> to continue, or q <return> to quit--- #27 0x00007f943f5a59bc in invoke (this=0x7f940c06fd50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:281 #28 connection_available (this=0x7f940c06ffe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:300 #29 on_connected (err=<optimized out>, sock_=<optimized out>, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:340 #30 on_connected (sock=<optimized out>, arg=<optimized out>, err=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/manager.cc:310 #31 0x00007f943f5a21f9 in handler (this=0x7f940c25a5b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/connect.cc:162 #32 lcb::io::Timer<lcb::io::Connstart, &lcb::io::Connstart::handler>::cb (arg=0x7f940c25a5b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/timer-cxx.h:80 #33 0x00007f943f59ecad in timer_callback (sock=<optimized out>, which=<optimized out>, arg=0x7f940c28b830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/lcbio/timer.c:45 #34 0x00007f943f5828f7 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/plugins/io/select/plugin-select.c:289 #35 0x00007f943f61147e in lcb_wait (instance=0x7f943432ce40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/wait.cc:103 #36 0x0000000000463596 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>, initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40 #37 0x000000000046581d in timer::TimerStore::Upsert (this=this@entry=0x7f9434322b30, key=..., value=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:362 #38 0x000000000046683a in timer::TimerStore::SetTimer (this=0x7f9434322b30, timer=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:72 #39 0x000000000041f846 in V8Worker::SetTimer (this=this@entry=0x7f94340b09a0, tinfo=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:1148 #40 0x000000000043dbdf in Timer::CreateTimerImpl (this=0x7f9434223020, args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:98 #41 0x000000000043e286 in CreateTimer (args=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:142 #42 0x00007f9440d65239 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) () from /opt/couchbase/lib/libv8.so #43 0x00007f9440d64738 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) () from /opt/couchbase/lib/libv8.so #44 0x00007f9440d63ec6 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) () from /opt/couchbase/lib/libv8.so #45 0x00007f944159d8ae in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit () from /opt/couchbase/lib/libv8.so #46 0x00002ce55018816e in ?? () #47 0x00002829aa1025a1 in ?? () #48 0x0000106b2a31d139 in ?? () #49 0x0000000900000000 in ?? () #50 0x00002829aa102681 in ?? () #51 0x00002fe24f4fee09 in ?? () #52 0x0000195cae456e91 in ?? () #53 0x00002fe24f4fed11 in ?? () #54 0x00002e9736e04ad1 in ?? () #55 0x00001e19ba08b1e1 in ?? () #56 0x00002fe24f4fee09 in ?? () #57 0x0000195cae456e91 in ?? () #58 0x00002fe24f4fed11 in ?? () #59 0x00002e9736e04ad1 in ?? () #60 0x0000106b2a31d139 in ?? () #61 0x00002fe24f4fee09 in ?? () #62 0x0000195cae456eb9 in ?? () #63 0x00002fe24f4fed11 in ?? () #64 0x000000cf00000000 in ?? () #65 0x00002e9736e04f59 in ?? () #66 0x00002e9736e04a69 in ?? () #67 0x0000106b2a302ad9 in ?? () #68 0x00007f9432ff9578 in ?? () ---Type <return> to continue, or q <return> to quit--- #69 0x00007f944150d603 in Builtins_JSEntryTrampoline () from /opt/couchbase/lib/libv8.so #70 0x00002fe24f4feca1 in ?? () #71 0x00001e19ba08b1e1 in ?? () #72 0x00002e9736e04a69 in ?? () #73 0x0000000000000020 in ?? () #74 0x00007f9432ff95e0 in ?? () #75 0x00002ce5501840de in ?? () #76 0x0000000000000000 in ?? () We see the first occurrence of Address 0x7f940c2f2c78 out of bounds at frame 21 - #21 0x00007f943f5bb60c in lcb::Authenticator::username_for[abi:cxx11](char const*, char const*, char const*, bool) (this=this@entry=0x7f943432f8b0, host=host@entry=0x7f940c2f2c78 <Address 0x7f940c2f2c78 out of bounds>, port=port@entry=0x7f940c2f307a <Address 0x7f940c2f307a out of bounds>, bucket=<optimized out>, use_cache=use_cache@entry=false) at /home/couchbase/jenkins/workspace/couchbase-server-unix/libcouchbase/src/auth.cc:90 I've attached the core dump for eventing-consumer - eventing-consumer-CB-4926.core Requesting SDK team to have a look and to assign back to us if the problem lies with eventing.

            Gautham Banasandra, it seems like the SDK issue, and indeed the hostname/port might come into this function while being invalid already. I will create CCBC ticket and link to this one.

            avsej Sergey Avseyev added a comment - Gautham Banasandra , it seems like the SDK issue, and indeed the hostname/port might come into this function while being invalid already. I will create CCBC ticket and link to this one.
            ingenthr Matt Ingenthron added a comment - - edited

            -Per discussion with Sergey Avseyev, ETA for fix CCBC-1131 (believed to be the root cause here) should be available today. - Vikas Chaudhary: do you want to verify the fix off of a toy build? If so, please have a look at the changes there. At the moment, all of the branches are locked (though the 2.10 should not be, something we're going to ask to be corrected).

            Update: the issue wasn't as obvious as initially identified. See Sergey's details below.

            ingenthr Matt Ingenthron added a comment - - edited -Per discussion with Sergey Avseyev , ETA for fix CCBC-1131 (believed to be the root cause here) should be available today. - Vikas Chaudhary : do you want to verify the fix off of a toy build? If so, please have a look at the changes there. At the moment, all of the branches are locked (though the 2.10 should not be, something we're going to ask to be corrected). Update: the issue wasn't as obvious as initially identified. See Sergey's details below.
            avsej Sergey Avseyev added a comment - - edited

            After more thorough look, I have noticed that we are copying the container of host/port by value here
            https://github.com/couchbase/libcouchbase/blob/201da089291a51b00c0198587c1a63f915f238f0/src/lcbio/connect.cc#L463
            So it should be okay, as we keep lifetime of the socket the same with its info object (which stores embedded host/port).

            These findings mean that I don't know why the address was invalidated, because the only place where we do it is the socket destructor.

            Here is my another suggestion:
            With this patch (already applied to branch release-2.10) http://review.couchbase.org/c/119085/2/cmake/Modules/GetLibcouchbaseFlags.cmake
            It is getting very easy to get a build with address-sanitized libcouchbase (requires libasan package installed on the box). All you need is to apply the following patch to tlm/CMakeLists.txt (i.e. pass -DLCB_USE_ASAN=ON to cmake when configuring libcouchbase build):

                    IF (BUILD_ENTERPRISE AND EXISTS "${PROJECT_SOURCE_DIR}/libcouchbase")
            +        SET (LCB_USE_ASAN ON CACHE BOOL "Enable address sanitizer" FORCE)
                     SET (LCB_NO_PLUGINS ON CACHE BOOL "Don't build plugins" FORCE)
                     SET (LCB_NO_TESTS ON CACHE BOOL "Don't run tests" FORCE)
            

            I believe if we run the test with sanitized libcouchbase, it will help to get more data for CCBC-1130

            avsej Sergey Avseyev added a comment - - edited After more thorough look, I have noticed that we are copying the container of host/port by value here https://github.com/couchbase/libcouchbase/blob/201da089291a51b00c0198587c1a63f915f238f0/src/lcbio/connect.cc#L463 So it should be okay, as we keep lifetime of the socket the same with its info object (which stores embedded host/port). These findings mean that I don't know why the address was invalidated, because the only place where we do it is the socket destructor. Here is my another suggestion: With this patch (already applied to branch release-2.10) http://review.couchbase.org/c/119085/2/cmake/Modules/GetLibcouchbaseFlags.cmake It is getting very easy to get a build with address-sanitized libcouchbase (requires libasan package installed on the box). All you need is to apply the following patch to tlm/CMakeLists.txt (i.e. pass -DLCB_USE_ASAN=ON to cmake when configuring libcouchbase build):        IF (BUILD_ENTERPRISE AND EXISTS "${PROJECT_SOURCE_DIR}/libcouchbase") +       SET (LCB_USE_ASAN ON CACHE BOOL "Enable address sanitizer" FORCE)        SET (LCB_NO_PLUGINS ON CACHE BOOL "Don't build plugins" FORCE)        SET (LCB_NO_TESTS ON CACHE BOOL "Don't run tests" FORCE) I believe if we run the test with sanitized libcouchbase, it will help to get more data for CCBC-1130

            Jeelan Poola: I know this probably isn't directly for you, but I'm unsure who to assign to and you can probably help. Can you assign to whomever might be able to get tests running under tsan? See Sergey Avseyev's notes above.

            ingenthr Matt Ingenthron added a comment - Jeelan Poola : I know this probably isn't directly for you, but I'm unsure who to assign to and you can probably help. Can you assign to whomever might be able to get tests running under tsan? See Sergey Avseyev 's notes above.

             Turns out that with gcc it still required to define environment variable LD_PRELOAD=/lib64/libasan.so.5, and this affect on how whole server behaves, as it does not seem to be easy to define this env variable just for eventing or use clang.

            We need to find another way to introspect the process. Can we make a call tomorrow in Europe time, so that we can try to attach gdb and see what's in it?

            avsej Sergey Avseyev added a comment -  Turns out that with gcc it still required to define environment variable LD_PRELOAD=/lib64/libasan.so.5 , and this affect on how whole server behaves, as it does not seem to be easy to define this env variable just for eventing or use clang. We need to find another way to introspect the process. Can we make a call tomorrow in Europe time, so that we can try to attach gdb and see what's in it?

            >> so that we can try to attach gdb and see what's in it?
            Sergey Avseyev We're not sure if it's easy to reproduce this issue as it was seen in a system test that was running for about 2 days. All we have is the core dump - eventing-consumer-CB-4926.core

            Gautham.Banasandra Gautham Banasandra (Inactive) added a comment - >> so that we can try to attach gdb and see what's in it? Sergey Avseyev We're not sure if it's easy to reproduce this issue as it was seen in a system test that was running for about 2 days. All we have is the core dump - eventing-consumer-CB-4926.core

            This is true Gautham Banasandra, but tools like these make it more likely that you will find the issue earlier in execution. No guarantees, of course.

            ingenthr Matt Ingenthron added a comment - This is true Gautham Banasandra , but tools like these make it more likely that you will find the issue earlier in execution. No guarantees, of course.

            Can we re-run the test with with patch to libcouchbase? http://review.couchbase.org/c/119163

            avsej Sergey Avseyev added a comment - Can we re-run the test with with patch to libcouchbase? http://review.couchbase.org/c/119163

            It appears that MB-37167 and MB-37197 may be related. See updates in CCBC-1130, which tracks the changes being made.

            ingenthr Matt Ingenthron added a comment - It appears that MB-37167 and MB-37197 may be related. See updates in CCBC-1130 , which tracks the changes being made.

            Hi Jeelan Poola and Suraj Naik. I've just uploaded another patch http://review.couchbase.org/119269, includes log entries to track lifetime and some internal state of the configuration objects. Hopefully this will reveal what is going wrong there.

            The messages are written on INFO level and include "MB-37197" substring.

            avsej Sergey Avseyev added a comment - Hi Jeelan Poola and Suraj Naik . I've just uploaded another patch http://review.couchbase.org/119269 , includes log entries to track lifetime and some internal state of the configuration objects. Hopefully this will reveal what is going wrong there. The messages are written on INFO level and include " MB-37197 " substring.
            suraj.naik Suraj Naik (Inactive) added a comment - - edited

            Sergey Avseyev

            I ran the test with the above patch. It produced a minidump. The logs for that are here - > http://qa.sc.couchbase.com/job/dev_testbed_blr3/209/artifact/logs/testrunner-19-Dec-12_06-38-40/*zip*/testrunner-19-Dec-12_06-38-40.zip

            In the above logs node 172.23.106.73 is the eventing node, it has the logs from your patch. Please check it out.

            The toy build link is http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/847/
            You can get the debug symbol from the above link.

            The testing was done on centos7.

            Here is the backtrace with this run:

            (gdb) bt
            #0  0x00007fb2811c9207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
            #1  0x00007fb2811ca8f8 in __GI_abort () at abort.c:90
            #2  0x00007fb28120bd27 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fb28131d678 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
            #3  0x00007fb281214489 in malloc_printerr (ar_ptr=0x7fb281559760 <main_arena>, ptr=<optimized out>, str=0x7fb28131d780 "double free or corruption (out)", action=3) at malloc.c:5004
            #4  _int_free (av=0x7fb281559760 <main_arena>, p=<optimized out>, have_lock=0) at malloc.c:3843
            #5  0x00007fb28287e1cd in mcreq_packet_done (pipeline=0x7fb2780fed50, pkt=pkt@entry=0x7fb258032a10) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mc/mcreq.c:780
            #6  0x00007fb2828e9708 in mcreq__pktflush_callback (p=0x7fb258032a10, hint=527, arg=0x7fb27cc21470) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mc/mcreq-flush-inl.h:68
            #7  0x00007fb282881dab in netbuf_end_flush2 (mgr=mgr@entry=0x7fb2780fed90, nflushed=nflushed@entry=527, callback=callback@entry=0x7fb2828e9660 <mcreq__pktflush_callback(void*, nb_SIZE, void*)>, lloff=lloff@entry=8,
                arg=arg@entry=0x7fb27cc21470) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/netbuf/netbuf.c:684
            #8  0x00007fb2828ee0f3 in mcreq_flush_done_ex (now=<optimized out>, expected=527, nflushed=527, pl=0x7fb2780fed50) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mc/mcreq-flush-inl.h:97
            #9  on_flush_done (ctx=<optimized out>, expected=527, actual=527) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:85
            #10 0x00007fb28288744b in E_put_ex (nb=527, niov=<optimized out>, iov=0x7fb27cc21500, ctx=0x7fb278123de0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:538
            #11 lcbio_ctx_put_ex (ctx=ctx@entry=0x7fb278123de0, iov=iov@entry=0x7fb27cc21500, niov=<optimized out>, nb=527) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:616
            #12 0x00007fb2828e9609 in on_flush_ready (ctx=0x7fb278123de0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:68
            #13 0x00007fb282885d32 in E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7fb278123de0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:322
            #14 0x00007fb28286b872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c:323
            #15 0x00007fb2828fb25e in lcb_wait (instance=0x7fb2780d0090) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc:103
            #16 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>,
                initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40
            #17 0x0000000000464bf1 in timer::TimerStore::GetCounter (this=this@entry=0x7fb2780eefb0, key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:319
            #18 0x00000000004665c0 in timer::TimerStore::SetTimer (this=0x7fb2780eefb0, timer=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:47
            #19 0x000000000041f936 in V8Worker::SetTimer (this=this@entry=0x7fb2780148a0, tinfo=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:1154
            #20 0x000000000043dccf in Timer::CreateTimerImpl (this=0x7fb2780c3470, args=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:98
            #21 0x000000000043e376 in CreateTimer (args=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc:142
            #22 0x00007fb28404f239 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) () from /opt/couchbase/lib/libv8.so
            #23 0x00007fb28404e738 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) () from /opt/couchbase/lib/libv8.so
            #24 0x00007fb28404dec6 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) () from /opt/couchbase/lib/libv8.so
            #25 0x00007fb2848878ae in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit () from /opt/couchbase/lib/libv8.so
            #26 0x000031b7e448816e in ?? ()
            #27 0x00001a4b0ea025a1 in ?? ()
            #28 0x00003f834591d139 in ?? ()
            #29 0x0000000900000000 in ?? ()
            #30 0x00001a4b0ea02681 in ?? ()
            #31 0x0000295296eb8159 in ?? ()
            #32 0x00003f834595b2e9 in ?? ()
            #33 0x0000295296eb80d9 in ?? ()
            #34 0x00002381dd037621 in ?? ()
            #35 0x00001ae7c2585649 in ?? ()
            #36 0x0000295296eb8159 in ?? ()
            #37 0x00003f834595b2e9 in ?? ()
            #38 0x0000295296eb80d9 in ?? ()
            #39 0x00002381dd037621 in ?? ()
            #40 0x00003f834591d139 in ?? ()
            #41 0x0000295296eb8159 in ?? ()
            #42 0x0000295296eb80d9 in ?? ()
            #43 0x0000008900000000 in ?? ()
            #44 0x00002381dd037f71 in ?? ()
            #45 0x00002381dd0374e9 in ?? ()
            #46 0x00003f8345902ad9 in ?? ()
            #47 0x00007fb27cc224c8 in ?? ()
            #48 0x00007fb2847f7603 in Builtins_JSEntryTrampoline () from /opt/couchbase/lib/libv8.so
            #49 0x0000295296eb8069 in ?? ()
            #50 0x0000295296eb7e11 in ?? ()
            #51 0x00001ae7c2585649 in ?? ()
            #52 0x00002381dd0374e9 in ?? ()
            #53 0x0000000000000020 in ?? ()
            #54 0x00007fb27cc22530 in ?? ()
            #55 0x000031b7e44840de in ?? ()
            ---Type <return> to continue, or q <return> to quit---
            #56 0x0000000000000000 in ?? ()
            

            suraj.naik Suraj Naik (Inactive) added a comment - - edited Sergey Avseyev I ran the test with the above patch. It produced a minidump. The logs for that are here - > http://qa.sc.couchbase.com/job/dev_testbed_blr3/209/artifact/logs/testrunner-19-Dec-12_06-38-40/*zip*/testrunner-19-Dec-12_06-38-40.zip In the above logs node 172.23.106.73 is the eventing node, it has the logs from your patch. Please check it out. The toy build link is http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/847/ You can get the debug symbol from the above link. The testing was done on centos7. Here is the backtrace with this run: (gdb) bt # 0 0x00007fb2811c9207 in __GI_raise (sig=sig @entry = 6 ) at ../nptl/sysdeps/unix/sysv/linux/raise.c: 55 # 1 0x00007fb2811ca8f8 in __GI_abort () at abort.c: 90 # 2 0x00007fb28120bd27 in __libc_message (do_abort=do_abort @entry = 2 , fmt=fmt @entry = 0x7fb28131d678 "*** Error in `%s': %s: 0x%s ***\n" ) at ../sysdeps/unix/sysv/linux/libc_fatal.c: 196 # 3 0x00007fb281214489 in malloc_printerr (ar_ptr= 0x7fb281559760 <main_arena>, ptr=<optimized out>, str= 0x7fb28131d780 "double free or corruption (out)" , action= 3 ) at malloc.c: 5004 # 4 _int_free (av= 0x7fb281559760 <main_arena>, p=<optimized out>, have_lock= 0 ) at malloc.c: 3843 # 5 0x00007fb28287e1cd in mcreq_packet_done (pipeline= 0x7fb2780fed50 , pkt=pkt @entry = 0x7fb258032a10 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mc/mcreq.c: 780 # 6 0x00007fb2828e9708 in mcreq__pktflush_callback (p= 0x7fb258032a10 , hint= 527 , arg= 0x7fb27cc21470 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mc/mcreq-flush-inl.h: 68 # 7 0x00007fb282881dab in netbuf_end_flush2 (mgr=mgr @entry = 0x7fb2780fed90 , nflushed=nflushed @entry = 527 , callback=callback @entry = 0x7fb2828e9660 <mcreq__pktflush_callback( void *, nb_SIZE, void *)>, lloff=lloff @entry = 8 , arg=arg @entry = 0x7fb27cc21470 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/netbuf/netbuf.c: 684 # 8 0x00007fb2828ee0f3 in mcreq_flush_done_ex (now=<optimized out>, expected= 527 , nflushed= 527 , pl= 0x7fb2780fed50 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mc/mcreq-flush-inl.h: 97 # 9 on_flush_done (ctx=<optimized out>, expected= 527 , actual= 527 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc: 85 # 10 0x00007fb28288744b in E_put_ex (nb= 527 , niov=<optimized out>, iov= 0x7fb27cc21500 , ctx= 0x7fb278123de0 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c: 538 # 11 lcbio_ctx_put_ex (ctx=ctx @entry = 0x7fb278123de0 , iov=iov @entry = 0x7fb27cc21500 , niov=<optimized out>, nb= 527 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c: 616 # 12 0x00007fb2828e9609 in on_flush_ready (ctx= 0x7fb278123de0 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc: 68 # 13 0x00007fb282885d32 in E_handler (sock=<optimized out>, which=<optimized out>, arg= 0x7fb278123de0 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c: 322 # 14 0x00007fb28286b872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c: 323 # 15 0x00007fb2828fb25e in lcb_wait (instance= 0x7fb2780d0090 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc: 103 # 16 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0 > (callable= @0x40a460 : {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait @plt >, isRetriable=<optimized out>, initial_delay_milliseconds= 200 , max_retry_count= 5 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h: 40 # 17 0x0000000000464bf1 in timer::TimerStore::GetCounter ( this = this @entry = 0x7fb2780eefb0 , key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc: 319 # 18 0x00000000004665c0 in timer::TimerStore::SetTimer ( this = 0x7fb2780eefb0 , timer=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc: 47 # 19 0x000000000041f936 in V8Worker::SetTimer ( this = this @entry = 0x7fb2780148a0 , tinfo=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc: 1154 # 20 0x000000000043dccf in Timer::CreateTimerImpl ( this = 0x7fb2780c3470 , args=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc: 98 # 21 0x000000000043e376 in CreateTimer (args=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/timer.cc: 142 # 22 0x00007fb28404f239 in v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) () from /opt/couchbase/lib/libv8.so # 23 0x00007fb28404e738 in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper< false >(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) () from /opt/couchbase/lib/libv8.so # 24 0x00007fb28404dec6 in v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) () from /opt/couchbase/lib/libv8.so # 25 0x00007fb2848878ae in Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit () from /opt/couchbase/lib/libv8.so # 26 0x000031b7e448816e in ?? () # 27 0x00001a4b0ea025a1 in ?? () # 28 0x00003f834591d139 in ?? () # 29 0x0000000900000000 in ?? () # 30 0x00001a4b0ea02681 in ?? () # 31 0x0000295296eb8159 in ?? () # 32 0x00003f834595b2e9 in ?? () # 33 0x0000295296eb80d9 in ?? () # 34 0x00002381dd037621 in ?? () # 35 0x00001ae7c2585649 in ?? () # 36 0x0000295296eb8159 in ?? () # 37 0x00003f834595b2e9 in ?? () # 38 0x0000295296eb80d9 in ?? () # 39 0x00002381dd037621 in ?? () # 40 0x00003f834591d139 in ?? () # 41 0x0000295296eb8159 in ?? () # 42 0x0000295296eb80d9 in ?? () # 43 0x0000008900000000 in ?? () # 44 0x00002381dd037f71 in ?? () # 45 0x00002381dd0374e9 in ?? () # 46 0x00003f8345902ad9 in ?? () # 47 0x00007fb27cc224c8 in ?? () # 48 0x00007fb2847f7603 in Builtins_JSEntryTrampoline () from /opt/couchbase/lib/libv8.so # 49 0x0000295296eb8069 in ?? () # 50 0x0000295296eb7e11 in ?? () # 51 0x00001ae7c2585649 in ?? () # 52 0x00002381dd0374e9 in ?? () # 53 0x0000000000000020 in ?? () # 54 0x00007fb27cc22530 in ?? () # 55 0x000031b7e44840de in ?? () ---Type < return > to continue , or q < return > to quit--- # 56 0x0000000000000000 in ?? ()

            Sergey Avseyev
            I ran the test with both of your patches(memory-leak fix and extra logging). Eventing node crashed and generated minidump. Node 172.23.106.73 is the eventing node. You can download logs from here:
            http://qa.sc.couchbase.com/job/dev_testbed_blr3/215/artifact/logs/testrunner-19-Dec-12_13-07-24/test_1/.

            Here is the toy build link for debug symbols: http://server.jenkins.couchbase.com/job/toy-unix-simple/851/.

            Stack trace:

            (gdb) bt
            #0  0x00007f20639a459c in free () from /usr/lib64/libc-2.17.so
            #1  0x00007f2064fffbdb in lcbvb_destroy (conf=0x7f2044048600) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/vbucket/vbucket.c:853
            #2  0x00007f206503510e in decref (this=0x7f204404ca10) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/clconfig.h:546
            #3  update (data=0x7f204402beb0 <Address 0x7f204402beb0 out of bounds>, host=<optimized out>, this=0x7f205c0f9960) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:212
            #4  lcb::clconfig::cccp_update (provider=provider@entry=0x7f205c0f9960, host=<optimized out>, data=data@entry=0x7f204402beb0 <Address 0x7f204402beb0 out of bounds>)
                at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:175
            #5  0x00007f2065075dbd in lcb::Server::handle_nmv (this=this@entry=0x7f205c110550, resinfo=..., oldpkt=oldpkt@entry=0x7f205c15cee0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:156
            #6  0x00007f2065078b19 in try_read (ior=0x7f205c000c18, ctx=0x7f205c000bd0, this=0x7f205c110550) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:401
            #7  on_read (ctx=0x7f205c000bd0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:465
            #8  0x00007f2065011dfc in invoke_read_cb (nb=15043, ctx=0x7f205c000bd0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:278
            #9  E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f205c000bd0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:307
            #10 0x00007f2064ff7872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c:323
            #11 0x00007f206508725e in lcb_wait (instance=0x7f205c0d0070) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc:103
            #12 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>,
                initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40
            #13 0x0000000000463b4d in timer::TimerStore::Get (this=0x7f205c0eef90, key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:421
            #14 0x0000000000468408 in timer::Iterator::GetNextTimer (this=this@entry=0x7f205aff9b60, tevent=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc:52
            #15 0x0000000000468bd2 in timer::Iterator::GetNext (this=this@entry=0x7f205aff9b60, tevent=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc:131
            #16 0x000000000042ad1b in V8Worker::RouteMessage (this=0x7f205c014880) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:488
            #17 0x00007f20644dadcf in std::execute_native_thread_routine (__p=0x7f205c0eb980) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83
            #18 0x00007f2063cf3dd5 in start_thread () from /usr/lib64/libpthread-2.17.so
            #19 0x00007f2063a1cead in clone () from /usr/lib64/libc-2.17.so
            

            satya.nand Satya Nand (Inactive) added a comment - Sergey Avseyev I ran the test with both of your patches(memory-leak fix and extra logging). Eventing node crashed and generated minidump. Node 172.23.106.73 is the eventing node. You can download logs from here: http://qa.sc.couchbase.com/job/dev_testbed_blr3/215/artifact/logs/testrunner-19-Dec-12_13-07-24/test_1/ . Here is the toy build link for debug symbols: http://server.jenkins.couchbase.com/job/toy-unix-simple/851/ . Stack trace: (gdb) bt # 0 0x00007f20639a459c in free () from /usr/lib64/libc- 2.17 .so # 1 0x00007f2064fffbdb in lcbvb_destroy (conf= 0x7f2044048600 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/vbucket/vbucket.c: 853 # 2 0x00007f206503510e in decref ( this = 0x7f204404ca10 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/clconfig.h: 546 # 3 update (data= 0x7f204402beb0 <Address 0x7f204402beb0 out of bounds>, host=<optimized out>, this = 0x7f205c0f9960 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc: 212 # 4 lcb::clconfig::cccp_update (provider=provider @entry = 0x7f205c0f9960 , host=<optimized out>, data=data @entry = 0x7f204402beb0 <Address 0x7f204402beb0 out of bounds>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc: 175 # 5 0x00007f2065075dbd in lcb::Server::handle_nmv ( this = this @entry = 0x7f205c110550 , resinfo=..., oldpkt=oldpkt @entry = 0x7f205c15cee0 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc: 156 # 6 0x00007f2065078b19 in try_read (ior= 0x7f205c000c18 , ctx= 0x7f205c000bd0 , this = 0x7f205c110550 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc: 401 # 7 on_read (ctx= 0x7f205c000bd0 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc: 465 # 8 0x00007f2065011dfc in invoke_read_cb (nb= 15043 , ctx= 0x7f205c000bd0 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c: 278 # 9 E_handler (sock=<optimized out>, which=<optimized out>, arg= 0x7f205c000bd0 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c: 307 # 10 0x00007f2064ff7872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c: 323 # 11 0x00007f206508725e in lcb_wait (instance= 0x7f205c0d0070 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc: 103 # 12 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0 > (callable= @0x40a460 : {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait @plt >, isRetriable=<optimized out>, initial_delay_milliseconds= 200 , max_retry_count= 5 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h: 40 # 13 0x0000000000463b4d in timer::TimerStore::Get ( this = 0x7f205c0eef90 , key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc: 421 # 14 0x0000000000468408 in timer::Iterator::GetNextTimer ( this = this @entry = 0x7f205aff9b60 , tevent=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc: 52 # 15 0x0000000000468bd2 in timer::Iterator::GetNext ( this = this @entry = 0x7f205aff9b60 , tevent=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc: 131 # 16 0x000000000042ad1b in V8Worker::RouteMessage ( this = 0x7f205c014880 ) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc: 488 # 17 0x00007f20644dadcf in std::execute_native_thread_routine (__p= 0x7f205c0eb980 ) at /tmp/deploy/gcc- 7.3 . 0 /libstdc++-v3/src/c++ 11 /thread.cc: 83 # 18 0x00007f2063cf3dd5 in start_thread () from /usr/lib64/libpthread- 2.17 .so # 19 0x00007f2063a1cead in clone () from /usr/lib64/libc- 2.17 .so

            Hi Satya Nand and Suraj Naik, I've just updated extra info patch. Could you rebuild toybuild and re-run the test?

            So far it there is not enough info to catch where the memory invalidated. The config object that carries the vbc instance we see in the stack trace, also created from NVM response, and in fact the library does not even use it

            2019-12-12T13:11:58.045-08:00 [lcb,server L:152 I:3174263777] <172.23.106.67:11210> (CTX=0x7f205c001b00,memcached,SRV=0x7f205c0fd9c0,IX=1) MB-37197. received NMV in response to op=0x00, opaque=0x377b, vbucket=504, node=172.23.106.67:11210, str=0x7f204402beb0
            2019-12-12T13:11:58.047-08:00 [lcb,cccp L:205 I:3174263777] <NOHOST:NOPORT> (CTX=(nil),) MB-37197. parsed and loaded config into struct cfg=0x7f204404ca10, vbc=0x7f2044048600, str=0x7f204402beb0, rev=114, oldcfg=0x7f204400f450
            2019-12-12T13:11:58.047-08:00 [lcb,cccp L:210 I:3174263777] MB-37197. [CccpProvider::update] decref oldcfg=0x7f204400f450 (rev=114), refcount=1
            2019-12-12T13:11:58.047-08:00 [lcb,confmon L:127 I:3174263777] MB-37197. notified about new config (notify_miss=1) cfg=0x7f204404ca10 (rev=114), oldcfg=0x7f20440434b0 (rev=115)
            2019-12-12T13:11:58.047-08:00 [lcb,confmon L:144 I:3174263777] MB-37197. compared configs cfg=0x7f204404ca10 (rev=114), oldcfg=0x7f20440434b0 (rev=115), res=0x02
            2019-12-12T13:11:58.047-08:00 [lcb,confmon L:153 I:3174263777] MB-37197. not applying config cfg=0x7f204404ca10 (rev=114), oldcfg=0x7f20440434b0 (rev=115)
            

            Next occurrence is couple seconds below

            2019-12-12T13:11:58.621-08:00 [lcb,server L:152 I:3174263777] <172.23.106.69:11210> (CTX=0x7f205c000bd0,memcached,SRV=0x7f205c110550,IX=2) MB-37197. received NMV in response to op=0x00, opaque=0x38f2, vbucket=851, node=172.23.106.69:11210, str=0x7f204402beb0
            2019-12-12T13:11:58.622-08:00 [lcb,cccp L:205 I:3174263777] <NOHOST:NOPORT> (CTX=(nil),) MB-37197. parsed and loaded config into struct cfg=0x7f2044008370, vbc=0x7f2044001dd0, str=0x7f204402beb0, rev=122, oldcfg=0x7f204404ca10
            2019-12-12T13:11:58.622-08:00 [lcb,cccp L:210 I:3174263777] MB-37197. [CccpProvider::update] decref oldcfg=0x7f204404ca10 (rev=114), refcount=1
            2019-12-12T13:11:58.688-08:00
            2019-12-12T13:11:58.688-08:00 == Minidump location: /opt/couchbase/var/lib/couchbase/crash/6638c4db-a241-669e-4d99a689-2c4f3758.dmp Status: 1 ==
            

            In the updated patch I also include in the log all invocations of lcbvb_destroy

            avsej Sergey Avseyev added a comment - Hi Satya Nand and Suraj Naik , I've just updated extra info patch. Could you rebuild toybuild and re-run the test? So far it there is not enough info to catch where the memory invalidated. The config object that carries the vbc instance we see in the stack trace, also created from NVM response, and in fact the library does not even use it 2019-12-12T13:11:58.045-08:00 [lcb,server L:152 I:3174263777] <172.23.106.67:11210> (CTX=0x7f205c001b00,memcached,SRV=0x7f205c0fd9c0,IX=1) MB-37197. received NMV in response to op=0x00, opaque=0x377b, vbucket=504, node=172.23.106.67:11210, str=0x7f204402beb0 2019-12-12T13:11:58.047-08:00 [lcb,cccp L:205 I:3174263777] <NOHOST:NOPORT> (CTX=(nil),) MB-37197. parsed and loaded config into struct cfg=0x7f204404ca10, vbc=0x7f2044048600, str=0x7f204402beb0, rev=114, oldcfg=0x7f204400f450 2019-12-12T13:11:58.047-08:00 [lcb,cccp L:210 I:3174263777] MB-37197. [CccpProvider::update] decref oldcfg=0x7f204400f450 (rev=114), refcount=1 2019-12-12T13:11:58.047-08:00 [lcb,confmon L:127 I:3174263777] MB-37197. notified about new config (notify_miss=1) cfg=0x7f204404ca10 (rev=114), oldcfg=0x7f20440434b0 (rev=115) 2019-12-12T13:11:58.047-08:00 [lcb,confmon L:144 I:3174263777] MB-37197. compared configs cfg=0x7f204404ca10 (rev=114), oldcfg=0x7f20440434b0 (rev=115), res=0x02 2019-12-12T13:11:58.047-08:00 [lcb,confmon L:153 I:3174263777] MB-37197. not applying config cfg=0x7f204404ca10 (rev=114), oldcfg=0x7f20440434b0 (rev=115) Next occurrence is couple seconds below 2019-12-12T13:11:58.621-08:00 [lcb,server L:152 I:3174263777] <172.23.106.69:11210> (CTX=0x7f205c000bd0,memcached,SRV=0x7f205c110550,IX=2) MB-37197. received NMV in response to op=0x00, opaque=0x38f2, vbucket=851, node=172.23.106.69:11210, str=0x7f204402beb0 2019-12-12T13:11:58.622-08:00 [lcb,cccp L:205 I:3174263777] <NOHOST:NOPORT> (CTX=(nil),) MB-37197. parsed and loaded config into struct cfg=0x7f2044008370, vbc=0x7f2044001dd0, str=0x7f204402beb0, rev=122, oldcfg=0x7f204404ca10 2019-12-12T13:11:58.622-08:00 [lcb,cccp L:210 I:3174263777] MB-37197. [CccpProvider::update] decref oldcfg=0x7f204404ca10 (rev=114), refcount=1 2019-12-12T13:11:58.688-08:00 2019-12-12T13:11:58.688-08:00 == Minidump location: /opt/couchbase/var/lib/couchbase/crash/6638c4db-a241-669e-4d99a689-2c4f3758.dmp Status: 1 == In the updated patch I also include in the log all invocations of lcbvb_destroy
            suraj.naik Suraj Naik (Inactive) added a comment - - edited

            Sergey Avseyev

            I ran the test again with this patch. It failed and the stack trace shows it failed in lcbvb_destroy - http://qa.sc.couchbase.com/job/dev_testbed_blr3/223/artifact/logs/testrunner-19-Dec-15_23-15-01/*zip*/testrunner-19-Dec-15_23-15-01.zip
            In the above logs, node 172.23.106.73 is the eventing node that contains the logs and the minidump.

            The toy build used to test this is http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/860/

            #0  __GI___libc_free (mem=0x7f234c423d70) at malloc.c:2941
            #1  0x00007f23772b1bfb in lcbvb_destroy (conf=0x7f234c023950) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/vbucket/vbucket.c:854
            #2  0x00007f23772e712e in decref (this=0x7f234c000e90) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/clconfig.h:546
            #3  update (data=0x7f234c0ae030 <Address 0x7f234c0ae030 out of bounds>, host=<optimized out>, this=0x7f23640f9990) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:212
            #4  lcb::clconfig::cccp_update (provider=provider@entry=0x7f23640f9990, host=<optimized out>, data=data@entry=0x7f234c0ae030 <Address 0x7f234c0ae030 out of bounds>)
                at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:175
            #5  0x00007f23773278cd in lcb::Server::handle_nmv (this=this@entry=0x7f23640fed50, resinfo=..., oldpkt=oldpkt@entry=0x7f236415d600) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:156
            #6  0x00007f237732a629 in try_read (ior=0x7f23640b2d88, ctx=0x7f23640b2d40, this=0x7f23640fed50) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:401
            #7  on_read (ctx=0x7f23640b2d40) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:465
            #8  0x00007f23772c3e1c in invoke_read_cb (nb=15044, ctx=0x7f23640b2d40) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:278
            #9  E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f23640b2d40) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:307
            #10 0x00007f23772a9872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c:323
            #11 0x00007f2377338d6e in lcb_wait (instance=0x7f23640d0090) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc:103
            #12 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>, 
                initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40
            #13 0x0000000000463b4d in timer::TimerStore::Get (this=0x7f23640eefb0, key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:421
            #14 0x0000000000468408 in timer::Iterator::GetNextTimer (this=this@entry=0x7f2371660b60, tevent=...)
                at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc:52
            #15 0x0000000000468bd2 in timer::Iterator::GetNext (this=this@entry=0x7f2371660b60, tevent=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc:131
            #16 0x000000000042ad1b in V8Worker::RouteMessage (this=0x7f23640148a0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:488
            #17 0x00007f237678cdcf in std::execute_native_thread_routine (__p=0x7f23640eb9a0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83
            #18 0x00007f2375fa5dd5 in start_thread (arg=0x7f2371664700) at pthread_create.c:307
            #19 0x00007f2375cceead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
            

            suraj.naik Suraj Naik (Inactive) added a comment - - edited Sergey Avseyev I ran the test again with this patch. It failed and the stack trace shows it failed in lcbvb_destroy - http://qa.sc.couchbase.com/job/dev_testbed_blr3/223/artifact/logs/testrunner-19-Dec-15_23-15-01/*zip*/testrunner-19-Dec-15_23-15-01.zip In the above logs, node 172.23.106.73 is the eventing node that contains the logs and the minidump. The toy build used to test this is http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/860/ #0 __GI___libc_free (mem=0x7f234c423d70) at malloc.c:2941 #1 0x00007f23772b1bfb in lcbvb_destroy (conf=0x7f234c023950) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/vbucket/vbucket.c:854 #2 0x00007f23772e712e in decref (this=0x7f234c000e90) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/clconfig.h:546 #3 update (data=0x7f234c0ae030 <Address 0x7f234c0ae030 out of bounds>, host=<optimized out>, this=0x7f23640f9990) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:212 #4 lcb::clconfig::cccp_update (provider=provider@entry=0x7f23640f9990, host=<optimized out>, data=data@entry=0x7f234c0ae030 <Address 0x7f234c0ae030 out of bounds>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:175 #5 0x00007f23773278cd in lcb::Server::handle_nmv (this=this@entry=0x7f23640fed50, resinfo=..., oldpkt=oldpkt@entry=0x7f236415d600) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:156 #6 0x00007f237732a629 in try_read (ior=0x7f23640b2d88, ctx=0x7f23640b2d40, this=0x7f23640fed50) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:401 #7 on_read (ctx=0x7f23640b2d40) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:465 #8 0x00007f23772c3e1c in invoke_read_cb (nb=15044, ctx=0x7f23640b2d40) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:278 #9 E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f23640b2d40) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:307 #10 0x00007f23772a9872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c:323 #11 0x00007f2377338d6e in lcb_wait (instance=0x7f23640d0090) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc:103 #12 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>, initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40 #13 0x0000000000463b4d in timer::TimerStore::Get (this=0x7f23640eefb0, key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:421 #14 0x0000000000468408 in timer::Iterator::GetNextTimer (this=this@entry=0x7f2371660b60, tevent=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc:52 #15 0x0000000000468bd2 in timer::Iterator::GetNext (this=this@entry=0x7f2371660b60, tevent=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_iterator.cc:131 #16 0x000000000042ad1b in V8Worker::RouteMessage (this=0x7f23640148a0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:488 #17 0x00007f237678cdcf in std::execute_native_thread_routine (__p=0x7f23640eb9a0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #18 0x00007f2375fa5dd5 in start_thread (arg=0x7f2371664700) at pthread_create.c:307 #19 0x00007f2375cceead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
            avsej Sergey Avseyev added a comment - - edited

            In the logs of latest run with more information in the log, I found interesting observation, the stack trace says we are crashing trying to deallocate `0x7f234c023950` as vbconf, and on the second frame we see that the wrapper-container has address `0x7f234c000e90`.  While in the logs I can see the following lines:  

            MB-37197. [~ConfigInfo] lcbvb_destroy conf=0x7fc38c000f50, vbc=0x7fc38c000e90 (rev=169)
            MB-37197. [lcbvb_destroy] free vbc=0x7fc38c000e90 (rev=169)
            

            And those are telling me the opposite, that `0x7fc38c000f50` is the wrapper, and vbconf has address `0x7fc38c000e90`. Which is correct, because the app continues working without any issue after these lines, and it is corresponds to what we output at confmon.cc:338  http://review.couchbase.org/c/119269/6/src/bucketconfig/confmon.cc#338

            Although it might be just reused addresses from allocator.

            Another, probably more interesting observation is that, the problem is not in freeing config itself, but rather its member:

                free(conf->buuid); 
            

            Initial value of buuid is NULL, because we use calloc to allocate whole structure. The only place we assign it is when we load config from JSON:

                if (get_jstr(cj, "uuid", &tmp)) {
                    cfg->buuid = strdup(tmp);
                }
            

            where get_jstr is defined as

            static int
            get_jstr(cJSON *parent, const char *key, char **value)
            {
                cJSON *res = cJSON_GetObjectItem(parent, key);
                if (res == NULL || res->type != cJSON_String) {
                    *value = NULL;
                    return 0;
                }
             
                *value = res->valuestring;
                return 1;
            }
            

            avsej Sergey Avseyev added a comment - - edited In the logs of latest run with more information in the log, I found interesting observation, the stack trace says we are crashing trying to deallocate `0x7f234c023950` as vbconf, and on the second frame we see that the wrapper-container has address `0x7f234c000e90`.  While in the logs I can see the following lines:   MB-37197. [~ConfigInfo] lcbvb_destroy conf=0x7fc38c000f50, vbc=0x7fc38c000e90 (rev=169) MB-37197. [lcbvb_destroy] free vbc=0x7fc38c000e90 (rev=169) And those are telling me the opposite, that `0x7fc38c000f50` is the wrapper, and vbconf has address `0x7fc38c000e90`. Which is correct, because the app continues working without any issue after these lines, and it is corresponds to what we output at confmon.cc:338   http://review.couchbase.org/c/119269/6/src/bucketconfig/confmon.cc#338 Although it might be just reused addresses from allocator. Another, probably more interesting observation is that, the problem is not in freeing config itself, but rather its member:     free(conf->buuid); Initial value of buuid is NULL, because we use calloc to allocate whole structure. The only place we assign it is when we load config from JSON: if (get_jstr(cj, "uuid", &tmp)) { cfg->buuid = strdup(tmp); } where get_jstr is defined as static int get_jstr(cJSON *parent, const char *key, char **value) { cJSON *res = cJSON_GetObjectItem(parent, key); if (res == NULL || res->type != cJSON_String) { *value = NULL; return 0; }   *value = res->valuestring; return 1; }
            avsej Sergey Avseyev added a comment - - edited

            So I've updated patch with tracing and nullifying buuid member of config structure here: http://review.couchbase.org/c/119269/6..7/src/vbucket/vbucket.c

            Satya Nand will run test again, and report results here

            avsej Sergey Avseyev added a comment - - edited So I've updated patch with tracing and nullifying buuid member of config structure here: http://review.couchbase.org/c/119269/6..7/src/vbucket/vbucket.c Satya Nand will run test again, and report results here

            Latest stack trace from Satya Nand

            (gdb) bt
            #0  __GI___libc_free (mem=0x7f238c40edd0) at malloc.c:2941
            #1  0x00007f23acaddc48 in lcbvb_destroy (conf=0x7f238c002e90) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/vbucket/vbucket.c:857
            #2  0x00007f23acb131be in decref (this=0x7f238c0113c0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/clconfig.h:546
            #3  update (data=0x7f238c094870 <Address 0x7f238c094870 out of bounds>, host=<optimized out>, this=0x7f239c0f9990) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:212
            #4  lcb::clconfig::cccp_update (provider=provider@entry=0x7f239c0f9990, host=<optimized out>, data=data@entry=0x7f238c094870 <Address 0x7f238c094870 out of bounds>)
                at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:175
            #5  0x00007f23acb5395d in lcb::Server::handle_nmv (this=this@entry=0x7f239c110580, resinfo=..., oldpkt=oldpkt@entry=0x7f239c138320) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:156
            #6  0x00007f23acb566b9 in try_read (ior=0x7f239c1296f8, ctx=0x7f239c1296b0, this=0x7f239c110580) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:401
            #7  on_read (ctx=0x7f239c1296b0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:465
            #8  0x00007f23acaefeac in invoke_read_cb (nb=15043, ctx=0x7f239c1296b0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:278
            #9  E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f239c1296b0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:307
            #10 0x00007f23acad5872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c:323
            #11 0x00007f23acb64dfe in lcb_wait (instance=0x7f239c0d0090) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc:103
            #12 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>,
                initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40
            #13 0x0000000000463b4d in timer::TimerStore::Get (this=this@entry=0x7f239c0eefb0, key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:421
            #14 0x0000000000465b82 in timer::TimerStore::SyncSpanLocked (this=this@entry=0x7f239c0eefb0, partition=partition@entry=695)
                at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:215
            #15 0x0000000000467641 in SyncSpan (this=0x7f239c0eefb0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:113
            #16 timer::TimerStore::GetIterator (this=0x7f239c0eefb0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:91
            #17 0x000000000042ac67 in V8Worker::RouteMessage (this=0x7f239c0148a0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:486
            #18 0x00007f23abfb8dcf in std::execute_native_thread_routine (__p=0x7f239c0eb9a0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83
            #19 0x00007f23ab7d1dd5 in start_thread (arg=0x7f23a27fc700) at pthread_create.c:307
            #20 0x00007f23ab4faead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
            

            http://qa.sc.couchbase.com/job/dev_testbed_blr3/229/artifact/logs/testrunner-19-Dec-16_09-14-38/test_1/

            avsej Sergey Avseyev added a comment - Latest stack trace from Satya Nand (gdb) bt #0 __GI___libc_free (mem=0x7f238c40edd0) at malloc.c:2941 #1 0x00007f23acaddc48 in lcbvb_destroy (conf=0x7f238c002e90) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/vbucket/vbucket.c:857 #2 0x00007f23acb131be in decref (this=0x7f238c0113c0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/clconfig.h:546 #3 update (data=0x7f238c094870 <Address 0x7f238c094870 out of bounds>, host=<optimized out>, this=0x7f239c0f9990) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:212 #4 lcb::clconfig::cccp_update (provider=provider@entry=0x7f239c0f9990, host=<optimized out>, data=data@entry=0x7f238c094870 <Address 0x7f238c094870 out of bounds>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/bucketconfig/bc_cccp.cc:175 #5 0x00007f23acb5395d in lcb::Server::handle_nmv (this=this@entry=0x7f239c110580, resinfo=..., oldpkt=oldpkt@entry=0x7f239c138320) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:156 #6 0x00007f23acb566b9 in try_read (ior=0x7f239c1296f8, ctx=0x7f239c1296b0, this=0x7f239c110580) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:401 #7 on_read (ctx=0x7f239c1296b0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/mcserver/mcserver.cc:465 #8 0x00007f23acaefeac in invoke_read_cb (nb=15043, ctx=0x7f239c1296b0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:278 #9 E_handler (sock=<optimized out>, which=<optimized out>, arg=0x7f239c1296b0) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/lcbio/ctx.c:307 #10 0x00007f23acad5872 in run_loop (io=<optimized out>, is_tick=<optimized out>) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/plugins/io/select/plugin-select.c:323 #11 0x00007f23acb64dfe in lcb_wait (instance=0x7f239c0d0090) at /home/couchbase/jenkins/workspace/toy-unix-simple/libcouchbase/src/wait.cc:103 #12 0x0000000000463686 in RetryWithFixedBackoff<bool (&)(lcb_error_t), lcb_error_t (&)(lcb_st*), lcb_st*&, lcb_error_t, 0> (callable=@0x40a460: {lcb_error_t (lcb_st *)} 0x40a460 <lcb_wait@plt>, isRetriable=<optimized out>, initial_delay_milliseconds=200, max_retry_count=5) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/features/include/retry_util.h:40 #13 0x0000000000463b4d in timer::TimerStore::Get (this=this@entry=0x7f239c0eefb0, key=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:421 #14 0x0000000000465b82 in timer::TimerStore::SyncSpanLocked (this=this@entry=0x7f239c0eefb0, partition=partition@entry=695) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:215 #15 0x0000000000467641 in SyncSpan (this=0x7f239c0eefb0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:113 #16 timer::TimerStore::GetIterator (this=0x7f239c0eefb0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing-ee/features/src/timer_store.cc:91 #17 0x000000000042ac67 in V8Worker::RouteMessage (this=0x7f239c0148a0) at /home/couchbase/jenkins/workspace/toy-unix-simple/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc:486 #18 0x00007f23abfb8dcf in std::execute_native_thread_routine (__p=0x7f239c0eb9a0) at /tmp/deploy/gcc-7.3.0/libstdc++-v3/src/c++11/thread.cc:83 #19 0x00007f23ab7d1dd5 in start_thread (arg=0x7f23a27fc700) at pthread_create.c:307 #20 0x00007f23ab4faead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 http://qa.sc.couchbase.com/job/dev_testbed_blr3/229/artifact/logs/testrunner-19-Dec-16_09-14-38/test_1/

            My tracepoints show that we are not trying to deallocate garbage-pointer, also there is only one strdup/free pair visible. So at the moment I don't know what to try except removing usage of this member (as we don't expose/use it anywhere yet)
            I did it in the patch here: http://review.couchbase.org/c/119269/7..8/src/vbucket/vbucket.c

            Satya Nand, Suraj Naik, could you rebuild toybuild and rerun the test, please?

            avsej Sergey Avseyev added a comment - My tracepoints show that we are not trying to deallocate garbage-pointer, also there is only one strdup/free pair visible. So at the moment I don't know what to try except removing usage of this member (as we don't expose/use it anywhere yet) I did it in the patch here: http://review.couchbase.org/c/119269/7..8/src/vbucket/vbucket.c Satya Nand , Suraj Naik , could you rebuild toybuild and rerun the test, please?

            Suraj Naik, Satya Nand, I've created clean version of the patch without any extra logging. Could you check it?

            http://review.couchbase.org/c/119474/

            While removing usage of buuid member fixes crashes, we still don't know what was the problem. So when the problem re-appear, please update the ticket.

            avsej Sergey Avseyev added a comment - Suraj Naik , Satya Nand , I've created clean version of the patch without any extra logging. Could you check it? http://review.couchbase.org/c/119474/ While removing usage of buuid member fixes crashes, we still don't know what was the problem. So when the problem re-appear, please update the ticket.

            Sergey Avseyev

            The toy build is ready with the above patch - http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/869/

            I have triggered a run with this toy build here - http://qa.sc.couchbase.com/job/dev_testbed_blr3/

            I would like to run this test quite a few times (>10). I will let you know the result after the runs are done.

            suraj.naik Suraj Naik (Inactive) added a comment - Sergey Avseyev The toy build is ready with the above patch - http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/869/ I have triggered a run with this toy build here - http://qa.sc.couchbase.com/job/dev_testbed_blr3/ I would like to run this test quite a few times (>10). I will let you know the result after the runs are done.

            Sergey Avseyev
            The test has passed in all the 10 runs with your patch in the testbed - http://qa.sc.couchbase.com/job/dev_testbed_blr3/ - run 240 to run 249. No crashes were observed in all the 10 runs.

            suraj.naik Suraj Naik (Inactive) added a comment - Sergey Avseyev The test has passed in all the 10 runs with your patch in the testbed - http://qa.sc.couchbase.com/job/dev_testbed_blr3/ - run 240 to run 249. No crashes were observed in all the 10 runs.
            dfinlay Dave Finlay added a comment -

            Thanks Suraj Naik. Sergey Avseyev, Matt Ingenthron: is there a libcouchbase patch that's ready to merge on the basis of these tests (that look to be successful)?

            dfinlay Dave Finlay added a comment - Thanks Suraj Naik . Sergey Avseyev , Matt Ingenthron : is there a libcouchbase patch that's ready to merge on the basis of these tests (that look to be successful)?

            It has been merged already: http://review.couchbase.org/c/119516/ this is the update for manifest

            avsej Sergey Avseyev added a comment - It has been merged already: http://review.couchbase.org/c/119516/ this is the update for manifest
            dfinlay Dave Finlay added a comment -

            Great! Thank you Sergey.

            dfinlay Dave Finlay added a comment - Great! Thank you Sergey.

            Not seen on 6.5.0-4960

            vikas.chaudhary Vikas Chaudhary added a comment - Not seen on 6.5.0-4960

            People

              suraj.naik Suraj Naik (Inactive)
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty