Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51105

[System Test] Caught unhandled std::exception-derived exception. what(): Monotonic<m> invariant failed: new value (7079) breaks invariant on current value (7094)

    XMLWordPrintable

Details

    Description

      QE TEST

      -test tests/eventing/neo/test_eventing_rebalance_rbac.yml -scope tests/eventing/neo/scope_eventing_rebalance.yml

      Day - 1
      Cycle - 1
      Scale - 3

      TEST STEP
      Rebalance in data node.

      [2022-02-20T12:20:08-08:00, sequoiatools/couchbase-cli:7.1:b5fe67] server-add -c 172.23.104.16:8091 --server-add https://172.23.97.77 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2022-02-20T12:20:18-08:00, sequoiatools/couchbase-cli:7.1:c50589] rebalance -c 172.23.104.16:8091 -u Administrator -p password
      		 
      →  
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.104.16:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs c50589
      		 
      docker start c50589
      		 
       
      		 
      *Unable to display progress bar on this os
      		 
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.

      CRASH
      On 172.23.97.77

      grep "CRITICAL" memcached.log       
      2022-02-20T12:29:17.551323-08:00 CRITICAL *** Fatal error encountered during exception handling ***
      		 
      2022-02-20T12:29:17.551385-08:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic<m> (ActiveStream eq_dcpq:eventing:G0KKOoPO-424:eventing:G0KKOoPO-423:worker_n1ql_op/scope_0/n1ql2_0_0_172.23.97.77:11210_172.23.121.165:8096/0 (vb:842) nextSnapStart) invariant failed: new value (7079) breaks invariant on current value (7094)
      		 
      2022-02-20T12:29:17.551394-08:00 CRITICAL Exception thrown from:
      		 
      2022-02-20T12:29:17.551438-08:00 CRITICAL     #0  /opt/couchbase/bin/memcached() [0x400000+0x13d0e2]
      		 
      2022-02-20T12:29:17.551455-08:00 CRITICAL     #1  /opt/couchbase/bin/memcached() [0x400000+0x4c04ec]
      		 
      2022-02-20T12:29:17.551469-08:00 CRITICAL     #2  /opt/couchbase/bin/memcached() [0x400000+0x4c9faf]
      		 
      2022-02-20T12:29:17.551486-08:00 CRITICAL     #3  /opt/couchbase/bin/memcached() [0x400000+0x4ca39c]
      		 
      2022-02-20T12:29:17.551497-08:00 CRITICAL     #4  /opt/couchbase/bin/memcached() [0x400000+0x4ca4b4]
      		 
      2022-02-20T12:29:17.551510-08:00 CRITICAL     #5  /opt/couchbase/bin/memcached() [0x400000+0x4d0b79]
      		 
      2022-02-20T12:29:17.551542-08:00 CRITICAL     #6  /opt/couchbase/bin/memcached() [0x400000+0x6b9c59]
      		 
      2022-02-20T12:29:17.551596-08:00 CRITICAL     #7  /opt/couchbase/bin/memcached() [0x400000+0x6b38ca]
      		 
      2022-02-20T12:29:17.551619-08:00 CRITICAL     #8  /opt/couchbase/bin/memcached() [0x400000+0x6bb00e]
      		 
      2022-02-20T12:29:17.551641-08:00 CRITICAL     #9  /opt/couchbase/bin/memcached() [0x400000+0x80f8b0]
      		 
      2022-02-20T12:29:17.551664-08:00 CRITICAL     #10 /opt/couchbase/bin/memcached() [0x400000+0x7fa34a]
      		 
      2022-02-20T12:29:17.551682-08:00 CRITICAL     #11 /opt/couchbase/bin/memcached() [0x400000+0x812869]
      		 
      2022-02-20T12:29:17.551702-08:00 CRITICAL     #12 /opt/couchbase/bin/memcached() [0x400000+0x6b35c4]
      		 
      2022-02-20T12:29:17.551743-08:00 CRITICAL     #13 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe25cfe1000+0xcdd40]
      		 
      2022-02-20T12:29:17.551757-08:00 CRITICAL     #14 /lib64/libpthread.so.0() [0x7fe25eeaf000+0x7dd5]
      		 
      2022-02-20T12:29:17.551846-08:00 CRITICAL     #15 /lib64/libc.so.6(clone+0x6d) [0x7fe25c6fa000+0xfdead]
      		 
      2022-02-20T12:29:18.475678-08:00 CRITICAL Detected previous crash
      		 
      2022-02-20T12:29:18.475732-08:00 CRITICAL Breakpad caught a crash (Couchbase version 7.1.0-2347). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/36fab46c-ed24-4fcd-acc2cebd-36df2eb9.dmp before terminating.
      		 
      2022-02-20T12:29:18.475742-08:00 CRITICAL Stack backtrace of crashed thread:
      		 
      2022-02-20T12:29:18.475743-08:00 CRITICAL    #0  /opt/couchbase/bin/memcached() [0x400000+0x740b08]
      		 
      2022-02-20T12:29:18.475744-08:00 CRITICAL    #1  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x7922da]
      		 
      2022-02-20T12:29:18.475745-08:00 CRITICAL    #2  /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x792618]
      		 
      2022-02-20T12:29:18.475746-08:00 CRITICAL    #3  /lib64/libpthread.so.0() [0x7fe25eeaf000+0xf5d0]
      		 
      2022-02-20T12:29:18.475747-08:00 CRITICAL    #4  /lib64/libc.so.6(gsignal+0x37) [0x7fe25c6fa000+0x36207]
      		 
      2022-02-20T12:29:18.475748-08:00 CRITICAL    #5  /lib64/libc.so.6(abort+0x148) [0x7fe25c6fa000+0x378f8]
      		 
      2022-02-20T12:29:18.475749-08:00 CRITICAL    #6  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe25cfe1000+0x9963c]
      		 
      2022-02-20T12:29:18.475749-08:00 CRITICAL    #7  /opt/couchbase/bin/memcached() [0x400000+0x74afdb]
      		 
      2022-02-20T12:29:18.475750-08:00 CRITICAL    #8  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe25cfe1000+0xa48f6]
      		 
      2022-02-20T12:29:18.475751-08:00 CRITICAL    #9  /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe25cfe1000+0xa4961]
      		 
      2022-02-20T12:29:18.475751-08:00 CRITICAL    #10 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe25cfe1000+0xa4bf4]
      		 
      2022-02-20T12:29:18.475752-08:00 CRITICAL    #11 /opt/couchbase/bin/memcached() [0x400000+0x13d1e7]
      		 
      2022-02-20T12:29:18.475753-08:00 CRITICAL    #12 /opt/couchbase/bin/memcached() [0x400000+0x4c04ec]
      		 
      2022-02-20T12:29:18.475753-08:00 CRITICAL    #13 /opt/couchbase/bin/memcached() [0x400000+0x4c9faf]
      		 
      2022-02-20T12:29:18.475754-08:00 CRITICAL    #14 /opt/couchbase/bin/memcached() [0x400000+0x4ca39c]
      		 
      2022-02-20T12:29:18.475755-08:00 CRITICAL    #15 /opt/couchbase/bin/memcached() [0x400000+0x4ca4b4]
      		 
      2022-02-20T12:29:18.475755-08:00 CRITICAL    #16 /opt/couchbase/bin/memcached() [0x400000+0x4d0b79]
      		 
      2022-02-20T12:29:18.475756-08:00 CRITICAL    #17 /opt/couchbase/bin/memcached() [0x400000+0x6b9c59]
      		 
      2022-02-20T12:29:18.475757-08:00 CRITICAL    #18 /opt/couchbase/bin/memcached() [0x400000+0x6b38ca]
      		 
      2022-02-20T12:29:18.475757-08:00 CRITICAL    #19 /opt/couchbase/bin/memcached() [0x400000+0x6bb00e]
      		 
      2022-02-20T12:29:18.475758-08:00 CRITICAL    #20 /opt/couchbase/bin/memcached() [0x400000+0x80f8b0]
      		 
      2022-02-20T12:29:18.475758-08:00 CRITICAL    #21 /opt/couchbase/bin/memcached() [0x400000+0x7fa34a]
      		 
      2022-02-20T12:29:18.475759-08:00 CRITICAL    #22 /opt/couchbase/bin/memcached() [0x400000+0x812869]
      		 
      2022-02-20T12:29:18.475760-08:00 CRITICAL    #23 /opt/couchbase/bin/memcached() [0x400000+0x6b35c4]
      		 
      2022-02-20T12:29:18.475761-08:00 CRITICAL    #24 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fe25cfe1000+0xcdd40]
      		 
      2022-02-20T12:29:18.475761-08:00 CRITICAL    #25 /lib64/libpthread.so.0() [0x7fe25eeaf000+0x7dd5]
      		 
      2022-02-20T12:29:18.475762-08:00 CRITICAL    #26 /lib64/libc.so.6(clone+0x6d) [0x7fe25c6fa000+0xfdead]

      NOTE
      Same crash was also observed on 7.1.0-2335 on another cluster.
      Last good build - 7.1.0-2314

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-51173 [System Test]Rebalance failed: (sid:77) nextSnapStart) invariant failed: new value (28306) breaks invariant on current value (28341

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-51260 [System Test] Timer handler stuck in deploying state

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-50543 [System Test] Seeing occurrences of "seq order violation" in projector logs for MAINT_STREAM

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-51298 Snapshot start seqno can be non Monotonic in the checkpoint manager

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sujay.gad Sujay Gad
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty