Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59282

[System Test] :- CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveStream::getOutstandingItems: stream:eq_dcpq:cbas:Local:bucket4:1c796c38c739683bc6dee69c0e89a438:2sid:5 vb:122

    XMLWordPrintable

Details

    • Untriaged
    • Linux x86_64
    • 0
    • Yes
    • Analytics Sprint 30

    Description

      Script to Repro

      ./sequoia -client 172.23.104.27:2375 -provider file:debian_pine.yml -test tests/integration/7.6/test_7.6.yml -scope tests/integration/7.6/scope_7.6_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.6.0-1692 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=1209600 -show_topology=true

      Few hours after starting the run we saw the follow CRITICAL messages.

      grep CRITICAL on 172.23.106.108

      root@s70601:/opt/couchbase/var/lib/couchbase/logs# grep CRITICAL *
      		 
      babysitter.log:[ns_server:info,2023-10-25T00:44:55.826-07:00,babysitter_of_ns_1@cb.local:<0.139.0>:ns_port_server:log:226]memcached<0.139.0>: 2023-10-25T00:44:55.624835-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      		 
      babysitter.log:memcached<0.139.0>: 2023-10-25T00:44:55.624900-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): ActiveStream::getOutstandingItems: stream:eq_dcpq:cbas:Local:bucket4:1c796c38c739683bc6dee69c0e89a438:2sid:5 vb:122 processing checkpoint type:InitialDisk, CheckpointHistorical::Yes, ranges:0, HCS:45, MVS:46, items:0
      		 
      babysitter.log:[ns_server:info,2023-10-25T00:44:56.661-07:00,babysitter_of_ns_1@cb.local:<0.139.0>:ns_port_server:log:226]memcached<0.139.0>: CRITICAL Breakpad caught a crash (Couchbase version 7.6.0-1692). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/6e461376-c1b4-479a-aea2dd81-b5f84aa1.dmp before terminating. Writing dump succeeded: yes
      		 
      babysitter.log:[ns_server:info,2023-10-25T00:45:29.980-07:00,babysitter_of_ns_1@cb.local:<0.19975.1>:ns_port_server:log:226]memcached<0.19975.1>: 2023-10-25T00:45:29.778883-07:00 CRITICAL Detected previous crash
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779117-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.6.0-1692). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/6e461376-c1b4-479a-aea2dd81-b5f84aa1.dmp before terminating. Writing dump succeeded: yes
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779146-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779149-07:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x8577d1]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779151-07:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x385) [0x400000+0x8751d5]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779152-07:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x9f) [0x400000+0x87551f]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779154-07:00 CRITICAL    #3  /lib/x86_64-linux-gnu/libc.so.6() [0x7fee73e1f000+0x3bfd0]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779156-07:00 CRITICAL    #4  /lib/x86_64-linux-gnu/libc.so.6() [0x7fee73e1f000+0x8ad3c]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779158-07:00 CRITICAL    #5  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x7fee73e1f000+0x3bf32]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779202-07:00 CRITICAL    #6  /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7fee73e1f000+0x26472]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779204-07:00 CRITICAL    #7  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fee71000000+0xa89ab]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779431-07:00 CRITICAL    #8  /opt/couchbase/bin/memcached() [0x400000+0x867416]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779434-07:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fee71000000+0xb82fa]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779436-07:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fee71000000+0xb8365]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779438-07:00 CRITICAL    #11 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fee71000000+0xb85b7]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779440-07:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x94b0c5]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779481-07:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0xe6ce4]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779484-07:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x5ccff3]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779630-07:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x5cd159]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779633-07:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x5d4051]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779635-07:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x7affaf]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779637-07:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x509ec2]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779639-07:00 CRITICAL    #19 /opt/couchbase/bin/memcached() [0x400000+0x7ae01a]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779640-07:00 CRITICAL    #20 /opt/couchbase/bin/memcached() [0x400000+0x7b1103]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779642-07:00 CRITICAL    #21 /opt/couchbase/bin/memcached() [0x400000+0x9470bf]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779644-07:00 CRITICAL    #22 /opt/couchbase/bin/memcached() [0x400000+0x92fbe7]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779646-07:00 CRITICAL    #23 /opt/couchbase/bin/memcached() [0x400000+0x949c5a]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779647-07:00 CRITICAL    #24 /opt/couchbase/bin/memcached() [0x400000+0x7aa360]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779650-07:00 CRITICAL    #25 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fee71000000+0xe4aa3]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779651-07:00 CRITICAL    #26 /lib/x86_64-linux-gnu/libc.so.6() [0x7fee73e1f000+0x89044]
      		 
      babysitter.log:memcached<0.19975.1>: 2023-10-25T00:45:29.779699-07:00 CRITICAL    #27 /lib/x86_64-linux-gnu/libc.so.6() [0x7fee73e1f000+0x1095fc]

      Similar stack traces were seen on 172.23.106.110 and 172.23.124.27 as well.
      This was not seen on the run we had 7.6.0-1668.

      Attachments

        1. bt.txt
          139 kB
        2. new_bt.txt
          160 kB

        Issue Links

          is triggering

          Bug - A problem which impairs or prevents the functions of the product. MB-59310 Handle empty snapshot from checkpoints

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty