Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40167

[Transaction DocIsolation]: Expects(bodySize == 0) fails in Item::removeUserXattrs()

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • No

    Description

      Build: 6.6.0-7845

      Scenario:

      1. Two node cluster, Couchbase bucket (replica=1)
      2. Create primary index over the couchbase-bucket
      3. Run Transaction (with durability=None) to insert 1 document

      Observation:

      Only when primary_index is created, Memcached crashes continuously on the active vbucket node and transaction fails with `com.couchbase.transactions.error.internal.AttemptExpired` exception.

      Without primary_index, same scenario works fine.

      Cbcollect logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/mc_crash/collectinfo-2020-06-26T130539-ns_1%4010.112.191.105.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/mc_crash/collectinfo-2020-06-26T130539-ns_1%4010.112.191.106.zip

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-40126 Subdoc CreateAsDeleted without any value path can result in non-empty value

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            drigby Dave Rigby added a comment - - edited

            Grepping for CRITICAL in .106 logs shows:

            2020-06-26T06:02:25.463510-07:00 INFO 61: (default) DCP (Producer) eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3 - (vb:971) ActiveStream::registerCursor name "eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3", backfill:true, seqno:3
            		 
            2020-06-26T06:02:25.463616-07:00 CRITICAL *** Fatal error encountered during exception handling ***
            		 
            2020-06-26T06:02:25.463646-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.cc: 480
            		 
            ...
            		 
            2020-06-26T06:02:25.487835-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.6.0-7845). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/51a0585f-c33d-ecc5-312b7470-097e5b60.dmp before terminating.
            		 
            2020-06-26T06:02:25.487850-07:00 CRITICAL Stack backtrace of crashed thread:
            		 
            2020-06-26T06:02:25.488176-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1352cd]
            		 
            2020-06-26T06:02:25.488227-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14d45e]
            		 
            2020-06-26T06:02:25.488272-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14d774]
            		 
            2020-06-26T06:02:25.488312-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7fa7d5dab000+0xf130]
            		 
            2020-06-26T06:02:25.488360-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x39) [0x7fa7d59ea000+0x355c9]
            		 
            2020-06-26T06:02:25.488403-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7fa7d59ea000+0x36cd8]
            		 
            2020-06-26T06:02:25.488448-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fa7d64e0000+0x91195]
            		 
            2020-06-26T06:02:25.488478-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x148d52]
            		 
            2020-06-26T06:02:25.488511-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8ef86]
            		 
            2020-06-26T06:02:25.488541-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8efd1]
            		 
            2020-06-26T06:02:25.488569-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8f213]
            		 
            2020-06-26T06:02:25.488591-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x13ff89]
            		 
            2020-06-26T06:02:25.488611-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x140138]
            		 
            2020-06-26T06:02:25.488628-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d4fd]
            		 
            2020-06-26T06:02:25.488643-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d947]
            		 
            2020-06-26T06:02:25.488657-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7d928]
            		 
            2020-06-26T06:02:25.488695-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1db972]
            		 
            2020-06-26T06:02:25.488701-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x1245a]
            		 
            2020-06-26T06:02:25.488704-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x111c6]
            		 
            2020-06-26T06:02:25.488707-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so(couchstore_changes_since+0x1d8) [0x7fa7d0c4e000+0x19fa8]
            		 
            2020-06-26T06:02:25.488712-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1dc897]
            		 
            2020-06-26T06:02:25.488715-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7eb12]
            		 
            2020-06-26T06:02:25.488718-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7ec58]
            		 
            2020-06-26T06:02:25.488721-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7c93a]
            		 
            2020-06-26T06:02:25.488723-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7cb8e]
            		 
            2020-06-26T06:02:25.488727-07:00 CRITICAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x12b6a4]
            		 
            2020-06-26T06:02:25.488729-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fa7d858f000+0x8f17]
            		 
            2020-06-26T06:02:25.488733-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7fa7d5dab000+0x7df3]
            		 
            2020-06-26T06:02:25.488756-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7fa7d59ea000+0xf61ad]

            The aforementioned failed expectation is related to MB-37374 (deleted user XATTRs):

            Item::removeUserXattrs()

            		 
                // The function currently does not support value with body.
            		 
                // That is fine for now as this is introduced for MB-37374, thus is supposed
            		 
                // to operate only against deleted items, which don't contain any body.
            		 
                Expects(isDeleted());
            		 
                const auto valNBytes = value->valueSize();
            		 
                cb::char_buffer valBuf{const_cast<char*>(value->getData()), valNBytes};
            		 
                const auto bodySize = valNBytes - cb::xattr::get_body_offset(valBuf);
            		 
                Expects(bodySize == 0);

            drigby Dave Rigby added a comment - - edited Grepping for CRITICAL in .106 logs shows: 2020-06-26T06:02:25.463510-07:00 INFO 61: (default) DCP (Producer) eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3 - (vb:971) ActiveStream::registerCursor name "eq_dcpq:secidx:proj-default-MAINT_STREAM_TOPIC_7c2cc7d3ecd849d461c04f3f1c8157a3-15896629439356753117/3", backfill:true, seqno:3 2020-06-26T06:02:25.463616-07:00 CRITICAL *** Fatal error encountered during exception handling *** 2020-06-26T06:02:25.463646-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): GSL: Precondition failure at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.cc: 480 ... 2020-06-26T06:02:25.487835-07:00 CRITICAL Breakpad caught a crash (Couchbase version 6.6.0-7845). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/51a0585f-c33d-ecc5-312b7470-097e5b60.dmp before terminating. 2020-06-26T06:02:25.487850-07:00 CRITICAL Stack backtrace of crashed thread: 2020-06-26T06:02:25.488176-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x1352cd] 2020-06-26T06:02:25.488227-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x14d45e] 2020-06-26T06:02:25.488272-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x14d774] 2020-06-26T06:02:25.488312-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fa7d5dab000+0xf130] 2020-06-26T06:02:25.488360-07:00 CRITICAL /lib64/libc.so.6(gsignal+0x39) [0x7fa7d59ea000+0x355c9] 2020-06-26T06:02:25.488403-07:00 CRITICAL /lib64/libc.so.6(abort+0x148) [0x7fa7d59ea000+0x36cd8] 2020-06-26T06:02:25.488448-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fa7d64e0000+0x91195] 2020-06-26T06:02:25.488478-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x148d52] 2020-06-26T06:02:25.488511-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8ef86] 2020-06-26T06:02:25.488541-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8efd1] 2020-06-26T06:02:25.488569-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fa7d64e0000+0x8f213] 2020-06-26T06:02:25.488591-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x13ff89] 2020-06-26T06:02:25.488611-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x140138] 2020-06-26T06:02:25.488628-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d4fd] 2020-06-26T06:02:25.488643-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x6d947] 2020-06-26T06:02:25.488657-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7d928] 2020-06-26T06:02:25.488695-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1db972] 2020-06-26T06:02:25.488701-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x1245a] 2020-06-26T06:02:25.488704-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so() [0x7fa7d0c4e000+0x111c6] 2020-06-26T06:02:25.488707-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/../lib/libcouchstore.so(couchstore_changes_since+0x1d8) [0x7fa7d0c4e000+0x19fa8] 2020-06-26T06:02:25.488712-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x1dc897] 2020-06-26T06:02:25.488715-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7eb12] 2020-06-26T06:02:25.488718-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7ec58] 2020-06-26T06:02:25.488721-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7c93a] 2020-06-26T06:02:25.488723-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x7cb8e] 2020-06-26T06:02:25.488727-07:00 CRITICAL /opt/couchbase/bin/../lib/../lib/ep.so() [0x7fa7d0ea5000+0x12b6a4] 2020-06-26T06:02:25.488729-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fa7d858f000+0x8f17] 2020-06-26T06:02:25.488733-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fa7d5dab000+0x7df3] 2020-06-26T06:02:25.488756-07:00 CRITICAL /lib64/libc.so.6(clone+0x6d) [0x7fa7d59ea000+0xf61ad] The aforementioned failed expectation is related to MB-37374 (deleted user XATTRs): Item::removeUserXattrs() // The function currently does not support value with body. // That is fine for now as this is introduced for MB-37374, thus is supposed // to operate only against deleted items, which don't contain any body. Expects(isDeleted()); const auto valNBytes = value->valueSize(); cb::char_buffer valBuf{ const_cast < char *>(value->getData()), valNBytes}; const auto bodySize = valNBytes - cb::xattr::get_body_offset(valBuf); Expects(bodySize == 0);
            drigby Dave Rigby added a comment -

            This is the same underlying issue as MB-40126, just manifesting in a slightly different way.

            drigby Dave Rigby added a comment - This is the same underlying issue as MB-40126 , just manifesting in a slightly different way.
            raju Raju Suravarjjala added a comment -

            Bulk closing invalid, duplicate and won't fix bugs

            raju Raju Suravarjjala added a comment - Bulk closing invalid, duplicate and won't fix bugs

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty