Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41024

Crash in DCP consumer when processing SyncWrite Prepare at end of snapshot under memory pressure

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes
    • KV Sprint 2020-July

    Description

      Crash is seen on node: 172.23.121.234

      Memcached logs:

      2020-08-16T15:29:52.970470-07:00 CRITICAL *** Fatal error encountered during exception handling ***
      2020-08-16T15:29:52.970522-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): Monotonic<15SnapshotEndInfo> (unlabelled) invariant failed: new value (3132{Memory}) breaks invariant on current value (3132{Memory})
      2020-08-16T15:29:53.149420-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2840). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/c0659945-f084-4b3a-ee89b6a2-5b140b54.dmp before terminating.
      2020-08-16T15:29:53.149450-07:00 CRITICAL Stack backtrace of crashed thread:
      2020-08-16T15:29:53.149642-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x13d9ed]
      2020-08-16T15:29:53.149661-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x153a2a]
      2020-08-16T15:29:53.149670-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x153d68]
      2020-08-16T15:29:53.149677-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f883512d000+0xf5f0]
      2020-08-16T15:29:53.149698-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7f8834d5f000+0x36337]
      2020-08-16T15:29:53.149717-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f8834d5f000+0x37a28]
      2020-08-16T15:29:53.149748-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7f8835862000+0x91195]
      2020-08-16T15:29:53.149757-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x14e482]
      2020-08-16T15:29:53.149773-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8835862000+0x8ef86]
      2020-08-16T15:29:53.149794-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8835862000+0x8efd1]
      2020-08-16T15:29:53.149808-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f8835862000+0x8f213]
      2020-08-16T15:29:53.149825-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0x106087]
      2020-08-16T15:29:53.149831-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0x1042a6]
      2020-08-16T15:29:53.149838-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0xcbf43]
      2020-08-16T15:29:53.149880-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0xd03b7]
      2020-08-16T15:29:53.149885-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0xd0568]
      2020-08-16T15:29:53.149889-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0xd1bbb]
      2020-08-16T15:29:53.149894-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0xb60ab]
      2020-08-16T15:29:53.149898-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0xb6229]
      2020-08-16T15:29:53.150043-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0xbaf28]
      2020-08-16T15:29:53.150050-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0x161b63]
      2020-08-16T15:29:53.150056-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7f8839009000+0x738ff]
      2020-08-16T15:29:53.150063-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f8837ae7000+0x10777]
      2020-08-16T15:29:53.150069-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7f883512d000+0x7e65]
      2020-08-16T15:29:53.150099-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f8834d5f000+0xfe88d]
      

      Test Details:
      root@172.23.96.248:/data/workspace/centos-p0-durability-vset00-00-tunable_rebalance_in_out_majority_6.5_P0/job_logs/testrunner-20-Aug-16_15-11-14/test_2

      QE test:

      num_items=100000,GROUP=P0;durability,EXCLUDE_GROUP=non_dgm;not_for_majority,durability=MAJORITY,active_resident_threshold=70,upgrade_version=7.0.0-2840 -t rebalance_new.rebalance_in_out.RebalanceInOutTests.test_incremental_rebalance_in_out_with_mutation,nodes_init=2,num_items=50000,value_size=512,replicas=2,GROUP=P0;MHbeta;durability;windows
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.0-3040 contains kv_engine commit b1dea0b with commit message:
            MB-41024: PassiveStream: Don't handleSnapshotEnd if message not processed

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-3040 contains kv_engine commit b1dea0b with commit message: MB-41024 : PassiveStream: Don't handleSnapshotEnd if message not processed
            ritam.sharma Ritam Sharma added a comment -

            James Harrison - Would you have specific functional tests to validate the issue. The issue was from Ritesh' tool to scan for dumps.

            ritam.sharma Ritam Sharma added a comment - James Harrison - Would you have specific functional tests to validate the issue. The issue was from Ritesh' tool to scan for dumps.

            Hi Ritam Sharma, the test is passing on 7.0.0-3583. Here is the job: test_2(http://qa.sc.couchbase.com/job/test_suite_executor-TAF/67226/console)

            ritesh.agarwal Ritesh Agarwal added a comment - Hi Ritam Sharma , the test is passing on 7.0.0-3583. Here is the job: test_2( http://qa.sc.couchbase.com/job/test_suite_executor-TAF/67226/console )
            ritam.sharma Ritam Sharma added a comment -

            Ritesh Agarwal - Thank you.

            ritam.sharma Ritam Sharma added a comment - Ritesh Agarwal - Thank you.

            Test passed on 6.6.1-9147 as well...

            ritesh.agarwal Ritesh Agarwal added a comment - Test passed on 6.6.1-9147 as well...

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty