Uploaded image for project: 'C++ Couchbase Client'
  1. C++ Couchbase Client
  2. CXXCBC-143

Segfault in bootstrap_handler when mcbp_session stopped while handling message

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 1.0.0-dp.13
    • 1.0.0-beta
    • None
    • 1

    Description

      http://sdk.jenkins.couchbase.com/job/c-cpp/job/cxx/job/centos-cxx-sdk-server-situational-tests/78/

      This was during a swap rebalance test

      It looks like the mcbp_session was stopped which resets the session_ ptr to null in the message handler while a message was being processed. There can be a race between the check here https://github.com/couchbaselabs/couchbase-cxx-client/blob/9d329392687fec35f3fa91c95da655bd4a70dacb/couchbase/io/mcbp_session.hxx#L193, session_ being reset here https://github.com/couchbaselabs/couchbase-cxx-client/blob/9d329392687fec35f3fa91c95da655bd4a70dacb/couchbase/io/mcbp_session.hxx#L121 and session_ being used later in the message handler

       

      13:24:17 Program terminated with signal SIGSEGV, Segmentation fault.
      13:24:17 #0  0x0000000000cdecdb in std::__atomic_base<unsigned int>::operator++ (this=0x538) at /usr/include/c++/8/bits/atomic_base.h:296
      13:24:17 296	      { return __atomic_add_fetch(&_M_i, 1, memory_order_seq_cst); }
      13:24:17 [Current thread is 1 (Thread 0x7fd01f6ba700 (LWP 30473))]
       
      13:24:20 Thread 1 (Thread 0x7fd01f6ba700 (LWP 30473)):
      13:24:20 #0  0x0000000000cdecdb in std::__atomic_base<unsigned int>::operator++ (this=0x538) at /usr/include/c++/8/bits/atomic_base.h:296
      13:24:20 #1  0x0000000000cc6b52 in couchbase::io::mcbp_session::next_opaque (this=0x0) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/couchbase/io/mcbp_session.hxx:923
      13:24:20 #2  0x0000000000cc0f8f in couchbase::io::mcbp_session::bootstrap_handler::handle (this=0x7fcf900044d0, msg=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/couchbase/io/mcbp_session.hxx:269
      13:24:20 #3  0x0000000000cc91a8 in couchbase::io::mcbp_session::do_read()::{lambda(std::error_code, unsigned long)#1}::operator()(std::error_code, unsigned long) const (__closure=0x7fcffc0166a0, ec=..., bytes_transferred=345) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/couchbase/io/mcbp_session.hxx:1191
      13:24:20 #4  0x0000000000cf320a in std::_Function_handler<void (std::error_code, unsigned long), couchbase::io::mcbp_session::do_read()::{lambda(std::error_code, unsigned long)#1}>::_M_invoke(std::_Any_data const&, std::error_code&&, unsigned long&&) (__functor=..., __args#0=..., __args#1=@0x7fd01f6b9170: 345) at /usr/include/c++/8/bits/std_function.h:297
      13:24:20 #5  0x0000000000d2dd51 in std::function<void (std::error_code, unsigned long)>::operator()(std::error_code, unsigned long) const (this=0x7fd01f6b9300, __args#0=..., __args#1=345) at /usr/include/c++/8/bits/std_function.h:687
      13:24:20 #6  0x0000000000d52f47 in asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>::operator()() (this=0x7fd01f6b9300) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/bind_handler.hpp:288
      13:24:20 #7  0x0000000000d4bbad in asio::asio_handler_invoke<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long> >(asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&, ...) (function=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/handler_invoke_hook.hpp:87
      13:24:20 #8  0x0000000000d437b5 in asio_handler_invoke_helpers::invoke<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>, std::function<void (std::error_code, unsigned long)> >(asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&, std::function<void (std::error_code, unsigned long)>&) (function=..., context=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/handler_invoke_helpers.hpp:54
      13:24:20 #9  0x0000000000d73ed9 in asio::detail::asio_handler_invoke<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>, std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>(asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&, asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>*) (function=..., this_handler=0x7fd01f6b9300) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/bind_handler.hpp:343
      13:24:20 #10 0x0000000000d7136e in asio_handler_invoke_helpers::invoke<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>, asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long> >(asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&, asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&) (function=..., context=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/handler_invoke_helpers.hpp:54
      13:24:20 #11 0x0000000000d6baf2 in asio::detail::executor_function::complete<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>, std::allocator<void> >(asio::detail::executor_function::impl_base*, bool) (base=0x7fcfdc005ce0, call=true) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/executor_function.hpp:115
      13:24:20 #12 0x0000000000caeb07 in asio::detail::executor_function::operator() (this=0x7fd01f6b94d8) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/executor_function.hpp:63
      13:24:20 #13 0x0000000000d5c1c1 in asio::asio_handler_invoke<asio::detail::executor_function> (function=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/handler_invoke_hook.hpp:87
      13:24:20 #14 0x0000000000d53669 in asio_handler_invoke_helpers::invoke<asio::detail::executor_function, asio::detail::executor_function> (function=..., context=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/handler_invoke_helpers.hpp:54
      13:24:20 #15 0x0000000000d5c368 in asio::detail::executor_op<asio::detail::executor_function, std::allocator<void>, asio::detail::scheduler_operation>::do_complete (owner=0x7fd00c03d110, base=0x7fd00003d7e0) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/executor_op.hpp:69
      13:24:20 #16 0x0000000000cafa4a in asio::detail::scheduler_operation::complete (this=0x7fd00003d7e0, owner=0x7fd00c03d110, ec=..., bytes_transferred=0) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/scheduler_operation.hpp:39
      13:24:20 #17 0x0000000000cb8fe6 in asio::detail::strand_executor_service::run_ready_handlers (impl=std::shared_ptr<asio::detail::strand_executor_service::strand_impl> (use count 5, weak count 0) = {...}) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/impl/strand_executor_service.ipp:149
      13:24:20 #18 0x0000000000d783f3 in asio::detail::strand_executor_service::invoker<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, void>::operator() (this=0x7fd01f6b9710) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/impl/strand_executor_service.hpp:126
      13:24:20 #19 0x0000000000d77419 in asio::asio_handler_invoke<asio::detail::strand_executor_service::invoker<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, void> > (function=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/handler_invoke_hook.hpp:87
      13:24:20 #20 0x0000000000d7566c in asio_handler_invoke_helpers::invoke<asio::detail::strand_executor_service::invoker<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, void>, asio::detail::strand_executor_service::invoker<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, void> > (function=..., context=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/handler_invoke_helpers.hpp:54
      13:24:20 #21 0x0000000000d73f64 in asio::io_context::basic_executor_type<std::allocator<void>, 4ul>::execute<asio::detail::strand_executor_service::invoker<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, void> > (this=0x7fd01f6b99e0, f=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/impl/io_context.hpp:287
      13:24:20 #22 0x0000000000d713f5 in asio_execution_execute_fn::impl::operator()<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const&, asio::detail::strand_executor_service::invoker<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, void> > (this=0x10442eb <asio_execution_execute_fn::static_instance<asio_execution_execute_fn::impl>::instance>, t=..., f=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/execution/execute.hpp:204
      13:24:20 #23 0x0000000000d6be80 in asio::detail::strand_executor_service::do_execute<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, asio::detail::executor_function, std::allocator<void> > (impl=std::shared_ptr<asio::detail::strand_executor_service::strand_impl> (use count 5, weak count 0) = {...}, ex=..., function=..., a=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/impl/strand_executor_service.hpp:257
      13:24:20 #24 0x0000000000d66033 in asio::detail::strand_executor_service::execute<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> const, asio::detail::executor_function> (impl=std::shared_ptr<asio::detail::strand_executor_service::strand_impl> (use count 5, weak count 0) = {...}, ex=..., function=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/impl/strand_executor_service.hpp:208
      13:24:20 #25 0x0000000000d5c0af in asio::strand<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> >::execute<asio::detail::executor_function> (this=0x7fd01f6b99e0, f=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/strand.hpp:286
      13:24:20 #26 0x0000000000d53503 in asio_execution_execute_fn::impl::operator()<asio::strand<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> > const&, asio::detail::executor_function> (this=0x10442eb <asio_execution_execute_fn::static_instance<asio_execution_execute_fn::impl>::instance>, t=..., f=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/execution/execute.hpp:204
      13:24:20 #27 0x0000000000d4c5a7 in asio::execution::detail::any_executor_base::execute_ex<asio::strand<asio::io_context::basic_executor_type<std::allocator<void>, 4ul> > > (ex=..., f=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/execution/any_executor.hpp:893
      13:24:20 #28 0x0000000000d5301b in asio::execution::detail::any_executor_base::execute<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long> >(asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&&) const (this=0x7fd01f6b99e0, f=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/execution/any_executor.hpp:610
      13:24:20 #29 0x0000000000d4bbd7 in asio_execution_execute_fn::impl::operator()<asio::any_io_executor, asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long> >(asio::any_io_executor&&, asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&&) const (this=0x10442eb <asio_execution_execute_fn::static_instance<asio_execution_execute_fn::impl>::instance>, t=..., f=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/execution/execute.hpp:204
      13:24:20 #30 0x0000000000d437ff in asio::detail::handler_work_base<asio::any_io_executor, void, asio::io_context, asio::executor, void>::dispatch<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>, std::function<void (std::error_code, unsigned long)> >(asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&, std::function<void (std::error_code, unsigned long)>&) (this=0x7fd01f6b9ae0, function=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/handler_work.hpp:438
      13:24:20 #31 0x0000000000d3bd41 in asio::detail::handler_work<std::function<void (std::error_code, unsigned long)>, asio::any_io_executor, void>::complete<asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long> >(asio::detail::binder2<std::function<void (std::error_code, unsigned long)>, std::error_code, unsigned long>&, std::function<void (std::error_code, unsigned long)>&) (this=0x7fd01f6b9ae0, function=..., handler=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/handler_work.hpp:515
      13:24:20 #32 0x0000000000d2e02f in asio::detail::reactive_socket_recv_op<asio::mutable_buffer, std::function<void (std::error_code, unsigned long)>, asio::any_io_executor>::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (owner=0x30c08d0, base=0x7fd00003d7e0) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/reactive_socket_recv_op.hpp:144
      13:24:20 #33 0x0000000000cafa4a in asio::detail::scheduler_operation::complete (this=0x7fd00003d7e0, owner=0x30c08d0, ec=..., bytes_transferred=0) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/scheduler_operation.hpp:39
      13:24:20 #34 0x0000000000cb2c5b in asio::detail::epoll_reactor::descriptor_state::do_complete (owner=0x30c08d0, base=0x7fcff8002820, ec=..., bytes_transferred=5) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/impl/epoll_reactor.ipp:805
      13:24:20 #35 0x0000000000cafa4a in asio::detail::scheduler_operation::complete (this=0x7fcff8002820, owner=0x30c08d0, ec=..., bytes_transferred=5) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/scheduler_operation.hpp:39
      13:24:20 #36 0x0000000000cb3c8c in asio::detail::scheduler::do_run_one (this=0x30c08d0, lock=..., this_thread=..., ec=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/impl/scheduler.ipp:491
      13:24:20 #37 0x0000000000cb3610 in asio::detail::scheduler::run (this=0x30c08d0, ec=...) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/detail/impl/scheduler.ipp:209
      13:24:20 #38 0x0000000000cb443b in asio::io_context::run (this=0x7ffedccca868) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/couchbase-cxx-client/third_party/asio/asio/include/asio/impl/io_context.ipp:62
      13:24:20 #39 0x0000000000c9866f in CBSdkd::MainDispatch::<lambda()>::operator()(void) const (__closure=0x30e9208) at /home/couchbase/jenkins/workspace/c-cpp/cxx/centos-cxx-sdk-server-situational-tests/sdkd-cxx/src/Control.cpp:16
      13:24:20 #40 0x0000000000c9bc85 in std::__invoke_impl<void, CBSdkd::MainDispatch::MainDispatch()::<lambda()> >(std::__invoke_other, CBSdkd::MainDispatch::<lambda()> &&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
      13:24:20 #41 0x0000000000c9ad0e in std::__invoke<CBSdkd::MainDispatch::MainDispatch()::<lambda()> >(CBSdkd::MainDispatch::<lambda()> &&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
      13:24:20 #42 0x0000000000cacb9c in std::thread::_Invoker<std::tuple<CBSdkd::MainDispatch::MainDispatch()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x30e9208) at /usr/include/c++/8/thread:244
      13:24:20 #43 0x0000000000cacb72 in std::thread::_Invoker<std::tuple<CBSdkd::MainDispatch::MainDispatch()::<lambda()> > >::operator()(void) (this=0x30e9208) at /usr/include/c++/8/thread:253
      13:24:20 #44 0x0000000000cacb56 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<CBSdkd::MainDispatch::MainDispatch()::<lambda()> > > >::_M_run(void) (this=0x30e9200) at /usr/include/c++/8/thread:196
      13:24:20 #45 0x00007fd02140fba3 in execute_native_thread_routine () from /lib64/libstdc++.so.6
      13:24:20 #46 0x00007fd02248714a in start_thread () from /lib64/libpthread.so.0
      13:24:20 #47 0x00007fd020aeadc3 in clone () from /lib64/libc.so.6
      

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              avsej Sergey Avseyev
              jake.rawsthorne Jake Rawsthorne
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes