Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57443

[System Test] : CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure: 'itemsForCursor.ranges.size() == 1' at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/engines/ep/src/dcp/active_stream.cc:1012

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 0
    • No

    Description

      Script to Repro

      ./sequoia -client 172.23.104.27:2375 -provider file:centos_pine.yml -test tests/integration/7.2/test_7.2.yml -scope tests/integration/7.2/scope_7.2_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.2.1-17234 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
      

      This is a Longevity run that has been running for past 3.5 days. It is based on a toy.
      http://server.jenkins.couchbase.com/view/Toys/job/toy-unix-simple/7234/parameters/

      Aditi Ahuja - Probably you can give context on the change you made in toy (probably irrelevant to this. Still a good idea I suppose).

      I see the same panic right around the same time on 4 different nodes. (172.23.108.103, 172.23.97.119, 172.23.97.122 and 172.23.99.21)

      bt of d151a1ac-8e55-4b88-3c91419c-c680a887.dmp on 172.23.108.103

      (gdb) bt
      #0  0x00007f87d9d8e207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
      #1  0x00007f87d9d8f8f8 in __GI_abort () at abort.c:90
      #2  0x00007f87da6d863c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95
      #3  0x0000000000b69f2b in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/utilities/terminate_handler.cc:88
      #4  0x00007f87da6e38f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48
      #5  0x00007f87da6e3961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58
      #6  0x00007f87da6e3bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f8758000940, tinfo=tinfo@entry=0xc7c5b8 <typeinfo for gsl::fail_fast>, dest=dest@entry=0x5a0440 <gsl::fail_fast::~fail_fast()>)
          at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95
      #7  0x0000000000451953 in gsl::detail::fail_fast_throw (message=0xcc3728 "GSL: Precondition failure: 'itemsForCursor.ranges.size() == 1' at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/engines/ep/src/dcp/active_stream.cc:1012")
          at /home/couchbase/jenkins/workspace/toy-unix-simple/third_party/gsl-lite/include/gsl/gsl-lite.hpp:1769
      #8  0x00000000004ec996 in ActiveStream::getOutstandingItems (this=0x7f85b0be3290, vb=...) at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/engines/ep/src/dcp/active_stream.cc:1025
      #9  0x00000000008fe2be in ActiveStream::nextCheckpointItemTask(std::lock_guard<std::mutex> const&) () at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/engines/ep/src/dcp/active_stream.cc:982
      #10 0x00000000008fe3f4 in ActiveStream::nextCheckpointItemTask() () at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/engines/ep/src/dcp/active_stream.cc:967
      #11 0x0000000000904a39 in ActiveStreamCheckpointProcessorTask::run() () at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/engines/ep/src/dcp/active_stream_checkpoint_processor_task.cc:52
      #12 0x0000000000ad86c9 in GlobalTask::execute(std::basic_string_view<char, std::char_traits<char> >) () at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/executor/globaltask.cc:98
      #13 0x0000000000ad1d8a in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#2}::operator()() const (__closure=0x7f877cfe7350) at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/executor/folly_executorpool.cc:309
      #14 0x0000000000ad9a7e in operator() (this=0x7f877cfe7350) at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/executor/cancellable_cpu_executor.cc:42
      #15 CancellableCPUExecutor::add(GlobalTask*, folly::Function<void ()>)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/executor/cancellable_cpu_executor.cc:42
      #16 0x0000000000c37a50 in operator() (this=0x7f877cfe7540) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416
      #17 folly::ThreadPoolExecutor::runTask(std::shared_ptr<folly::ThreadPoolExecutor::Thread> const&, folly::ThreadPoolExecutor::Task&&) (this=this@entry=0x7f87d858bd00, thread=<error reading variable: Cannot access memory at address 0x7f87d965f8b8>, 
          task=task@entry=<unknown type in /usr/lib/debug/opt/couchbase/bin/memcached-7.2.1-17234.x86_64.debug, CU 0xac1bf97, DIE 0xac9fecc>)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97
      #18 0x0000000000c224ea in folly::CPUThreadPoolExecutor::threadRun (this=0x7f87d858bd00, thread=<error reading variable: Cannot access memory at address 0x7f87d965f8b8>)
          at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265
      #19 0x0000000000c3aa09 in __invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__t=<optimized out>, 
          __f=<optimized out>) at /usr/local/include/c++/7.3.0/bits/invoke.h:73
      #20 __invoke<void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&> (__fn=<optimized out>)
          at /usr/local/include/c++/7.3.0/bits/invoke.h:95
      #21 __call<void, 0, 1> (__args=<optimized out>, this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:467
      #22 operator()<> (this=<optimized out>) at /usr/local/include/c++/7.3.0/functional:551
      #23 folly::detail::function::FunctionTraits<void ()>::callBig<std::_Bind<void (folly::ThreadPoolExecutor::*(folly::ThreadPoolExecutor*, std::shared_ptr<folly::ThreadPoolExecutor::Thread>))(std::shared_ptr<folly::ThreadPoolExecutor::Thread>)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401
      #24 0x0000000000ad1a84 in operator() (this=0x7f87d84a1380) at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/executor/folly_executorpool.cc:49
      #25 operator() (__closure=0x7f87d84a1380) at /home/couchbase/jenkins/workspace/toy-unix-simple/kv_engine/executor/folly_executorpool.cc:49
      #26 folly::detail::function::FunctionTraits<void ()>::callBig<CBRegisteredThreadFactory::newThread(folly::Function<void ()>&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...)
          at /home/couchbase/jenkins/workspace/toy-unix-simple/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401
      #27 0x00007f87da70cd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80
      #28 0x00007f87dc50ddd5 in start_thread (arg=0x7f877cff9700) at pthread_create.c:307
      #29 0x00007f87d9e55ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      (gdb) 
      

      cbcollect_info attached.

      Attachments

        1. bt_full.txt
          16 kB
        2. info_threads.txt
          8 kB
        3. thread_apply_all_bt.txt
          3 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty