Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48010

[System Test] memcached crash observed in longevity - Monotonic<m> (ActiveStream(vb:769)::lastReadSeqno) invariant failed: new value (25644) breaks invariant on current value (25644)

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes
    • KV-Engine Sprint 2021 August

    Description

      7.0.2-6522

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale 3
      1st iteration

      Following crash observed:

      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155358-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155431-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic<m> (ActiveStream(vb:769)::lastReadSeqno) invariant failed: new value (25644) breaks invariant on current value (25644)
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155444-07:00 CRITICAL Exception thrown from:
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155495-07:00 CRITICAL     #0  /opt/couchbase/bin/memcached() [0x400000+0xb2b90]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155536-07:00 CRITICAL     #1  /opt/couchbase/bin/memcached() [0x400000+0xce10c]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155594-07:00 CRITICAL     #2  /opt/couchbase/bin/memcached() [0x400000+0x481279]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155616-07:00 CRITICAL     #3  /opt/couchbase/bin/memcached() [0x400000+0x484207]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155637-07:00 CRITICAL     #4  /opt/couchbase/bin/memcached() [0x400000+0x4849cc]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155678-07:00 CRITICAL     #5  /opt/couchbase/bin/memcached() [0x400000+0x484ae4]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155358-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155431-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic<m> (ActiveStream(vb:769)::lastReadSeqno) invariant failed: new value (25644) breaks invariant on current value (25644)
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155444-07:00 CRITICAL Exception thrown from:
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155495-07:00 CRITICAL     #0  /opt/couchbase/bin/memcached() [0x400000+0xb2b90]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155536-07:00 CRITICAL     #1  /opt/couchbase/bin/memcached() [0x400000+0xce10c]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155594-07:00 CRITICAL     #2  /opt/couchbase/bin/memcached() [0x400000+0x481279]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155616-07:00 CRITICAL     #3  /opt/couchbase/bin/memcached() [0x400000+0x484207]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155637-07:00 CRITICAL     #4  /opt/couchbase/bin/memcached() [0x400000+0x4849cc]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155678-07:00 CRITICAL     #5  /opt/couchbase/bin/memcached() [0x400000+0x484ae4]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.155693-07:00 CRITICAL     #6  /opt/couchbase/bin/memcached() [0x400000+0x490219]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156331-07:00 CRITICAL     #7  /opt/couchbase/bin/memcached() [0x400000+0x2a6e57]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156377-07:00 CRITICAL     #8  /opt/couchbase/bin/memcached() [0x400000+0x2a1fa0]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156405-07:00 CRITICAL     #9  /opt/couchbase/bin/memcached() [0x400000+0x711f30]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156433-07:00 CRITICAL     #10 /opt/couchbase/bin/memcached() [0x400000+0x6f9cea]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156467-07:00 CRITICAL     #11 /opt/couchbase/bin/memcached() [0x400000+0x714ee9]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156487-07:00 CRITICAL     #12 /opt/couchbase/bin/memcached() [0x400000+0x2a067d]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156603-07:00 CRITICAL     #13 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xcdd40]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156612-07:00 CRITICAL     #14 /lib64/libpthread.so.0() [0x7f4680ff3000+0x7e65]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:14.156655-07:00 CRITICAL     #15 /lib64/libc.so.6(clone+0x6d) [0x7f467ea61000+0xfe88d]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.341814-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.2-6522). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/99b3493f-b170-4ea4-4d65f9be-9a4ca054.dmp before terminating.
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.341858-07:00 CRITICAL Stack backtrace of crashed thread:
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342083-07:00 CRITICAL     #0  /opt/couchbase/bin/memcached() [0x400000+0x676753]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342139-07:00 CRITICAL     #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x69776a]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342183-07:00 CRITICAL     #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x697aa8]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342194-07:00 CRITICAL     #3  /lib64/libpthread.so.0() [0x7f4680ff3000+0xf5f0]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342242-07:00 CRITICAL     #4  /lib64/libc.so.6(gsignal+0x37) [0x7f467ea61000+0x36337]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342286-07:00 CRITICAL     #5  /lib64/libc.so.6(abort+0x148) [0x7f467ea61000+0x37a28]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342358-07:00 CRITICAL     #6  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0x9963c]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342396-07:00 CRITICAL     #7  /opt/couchbase/bin/memcached() [0x400000+0x68438f]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342416-07:00 CRITICAL     #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xa48f6]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342449-07:00 CRITICAL     #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xa4961]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342483-07:00 CRITICAL     #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xa4bf4]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342499-07:00 CRITICAL     #11 /opt/couchbase/bin/memcached() [0x400000+0xb2c95]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342528-07:00 CRITICAL     #12 /opt/couchbase/bin/memcached() [0x400000+0xce10c]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342620-07:00 CRITICAL     #13 /opt/couchbase/bin/memcached() [0x400000+0x481279]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342676-07:00 CRITICAL     #14 /opt/couchbase/bin/memcached() [0x400000+0x484207]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342773-07:00 CRITICAL     #15 /opt/couchbase/bin/memcached() [0x400000+0x4849cc]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342804-07:00 CRITICAL     #16 /opt/couchbase/bin/memcached() [0x400000+0x484ae4]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342837-07:00 CRITICAL     #17 /opt/couchbase/bin/memcached() [0x400000+0x490219]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342876-07:00 CRITICAL     #18 /opt/couchbase/bin/memcached() [0x400000+0x2a6e57]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342949-07:00 CRITICAL     #19 /opt/couchbase/bin/memcached() [0x400000+0x2a1fa0]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.342978-07:00 CRITICAL     #20 /opt/couchbase/bin/memcached() [0x400000+0x711f30]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343005-07:00 CRITICAL     #21 /opt/couchbase/bin/memcached() [0x400000+0x6f9cea]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343028-07:00 CRITICAL     #22 /opt/couchbase/bin/memcached() [0x400000+0x714ee9]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343046-07:00 CRITICAL     #23 /opt/couchbase/bin/memcached() [0x400000+0x2a067d]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343087-07:00 CRITICAL     #24 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f467f349000+0xcdd40]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343100-07:00 CRITICAL     #25 /lib64/libpthread.so.0() [0x7f4680ff3000+0x7e65]
      		 
      /opt/couchbase/var/lib/couchbase/logs/memcached.log.000047.txt:2021-08-17T20:46:15.343630-07:00 CRITICAL     #26 /lib64/libc.so.6(clone+0x6d) [0x7f467ea61000+0xfe88d]

      At the same time, the following service exit is observed as well:

      172.23.120.73 : crash
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-08-17T20:49:09.734-07:00,ns_1@172.23.120.73:<0.21752.0>:ns_log:crash_consumption_loop:63]Service 'memcached' exited with status 134. Restarting. Messages:
      		 
       
      		 
      172.23.120.77 : crash
      		 
      /opt/couchbase/var/lib/couchbase/logs/info.log.4.gz:[user:info,2021-08-17T20:48:58.097-07:00,ns_1@172.23.120.77:<0.13228.0>:ns_log:crash_consumption_loop:63]Service 'memcached' exited with status 134. Restarting. Messages:

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.134.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.106.136.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.75.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.105.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.110.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.149.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.150.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1629299013/collectinfo-2021-08-18T150334-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-48358 [System Test] Rebalance exited due to memcached crash

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty