Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Duplicate
Priority: Major
Fix Version/s: None
Affects Version/s: Cheshire-Cat
Component/s: couchbase-bucket
Labels:
- jepsen
Environment:
Ubuntu 18.04.4 LTS
Couchbase version 7.0.0-2352

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-39950/039ce937/20200612T182532.000-0700.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Multiple crashes found during kv-engine-cheshire-cat-jepsen-nightly-166 they all seem to be due to the same exception being thrown Caught unhandled std::exception-derived exception. what(): ThrowExceptionUnderflowPolicy current:0 arg:-94. As backtraces all seem to have the same address offsets.
Jepsen test: lein trampoline run test --nodes-file ./nodes --username root --password couchbase --workload=kill --scenario=kill-memcached --replicas=2 --no-autofailover --durability=0:0:0:100 --eviction-policy=value --use-json-docs --doc-padding-size=4096 --enable-tcp-capture --hashdump --enable-memcached-debug-log-level --disable-out-of-order-execution --net-interface=eth0

 gdb /opt/couchbase/bin/memcached -c 3ab32dce-d729-42ba-d8a4809a-8ba7c02d.dmp.core

(gdb) bt

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

#1  0x00007f1ea8c8b801 in __GI_abort () at abort.c:79

#2  0x00007f1ea98a9165 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95

#3  0x000000000054cf72 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86

#4  0x00007f1ea98a6f56 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47

#5  0x00007f1ea98a6fa1 in std::terminate () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57

#6  0x00007f1ea98a71e3 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f1e54004b10, tinfo=0x7f1ea9b8aad8 <typeinfo for std::underflow_error>, dest=0x7f1ea98bc460 <std::underflow_error::~underflow_error()>)

    at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:93

#7  0x00007f1eacfd0f14 in cb::ThrowExceptionUnderflowPolicy<unsigned long>::underflow (this=<optimized out>, desired=<optimized out>, current=<optimized out>, arg=<optimized out>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:48

#8  0x00007f1ead1a4ade in cb::NonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::fetch_add (arg=-94, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:125

#9  cb::NonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::operator+= (rhs=18446744073709551522, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:182

#10 Collections::VB::ManifestEntry::updateDiskSize (delta=-94, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_entry.h:107

#11 Collections::VB::Manifest::updateDiskSize (delta=-94, entry=..., this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest.h:1055

#12 Collections::VB::Manifest::CachingReadHandle::updateDiskSize (delta=-94, this=0x7f1e6e7f1290) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest.h:393

#13 Collections::VB::Flush::updateDiskSize (this=<optimized out>, key=..., delta=-94) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/flush.cc:58

#14 0x00007f1ead16c3c2 in saveDocsCallback (oldInfo=0x7f1e6b12ce80, newInfo=0x7f1e6b12cef0, context=0x7f1e6e7f1a10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:2113

#15 0x00007f1eacd2733b in do_save_callback (valueNew=<optimized out>, valueOld=0x7f1e6e7f1440, key=0x7f1e6b01c540, rq=0x7f1e6e7f1610) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:431

#16 modify_node (rq=rq@entry=0x7f1e6e7f1610, nptr=nptr@entry=0x7f1e6b17c2c0, start=start@entry=0, end=1, dst=dst@entry=0x7f1e6a71d010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:537

#17 0x00007f1eacd27ca9 in modify_btree (rq=rq@entry=0x7f1e6e7f1610, root=0x7f1e6b17c2c0, errcode=errcode@entry=0x7f1e6e7f1544) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:809

#18 0x00007f1eacd3bd70 in update_indexes (save_callback_ctx=<optimized out>, save_callback=<optimized out>, numdocs=<optimized out>, idvals=<optimized out>, ids=<optimized out>, seqvals=<optimized out>, seqs=0x7f1e6b01c530, db=<optimized out>, this=<optimized out>,

    this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:238

#19 couchstore_save_documents_and_callback (db=<optimized out>, docs=<optimized out>, infos=<optimized out>, numdocs=<optimized out>, options=options@entry=3, save_cb=save_cb@entry=0x7f1ead16c2b0 <saveDocsCallback(DocInfo const*, DocInfo const*, void*)>,

    save_cb_ctx=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:424

#20 0x00007f1ead1784e8 in CouchKVStore::saveDocs (this=this@entry=0x7f1e9deb3000, vbid=..., docs=..., docinfos=..., kvctx=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:2163

#21 0x00007f1ead17941a in CouchKVStore::commit2couchstore (this=0x7f1e9deb3000, commitData=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:2018

#22 0x00007f1ead179ab5 in CouchKVStore::commit (this=0x7f1e9deb3000, commitData=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:1297

#23 0x00007f1ead0634c2 in EPBucket::commit (this=this@entry=0x7f1e9de9f000, vbid=..., kvstore=..., commitData=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:875

#24 0x00007f1ead066ff9 in EPBucket::flushVBucket (this=0x7f1e9de9f000, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:744

#25 0x00007f1ead0bbf6c in Flusher::flushVB (this=this@entry=0x7f1e9deec700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:306

#26 0x00007f1ead0bd139 in Flusher::step (this=0x7f1e9deec700, task=0x7f1e9de57110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:207

#27 0x00007f1ead0c00e3 in GlobalTask::execute (this=0x7f1e9de57110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:73

#28 0x00007f1ead0b68ff in ExecutorThread::run (this=0x7f1ea5617500) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:188

#29 0x00007f1eabab4387 in CouchbaseThread::run (this=0x7f1ea863d010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58

#30 platform_thread_wrap (arg=0x7f1ea863d010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71

#31 0x00007f1ea90436db in start_thread (arg=0x7f1e6e7fc700) at pthread_create.c:463

#32 0x00007f1ea8d6c88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

(gdb) bt full

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

        set = {__val = {0, 139769661323011, 139768235051680, 0, 139769661327536, 139768679565984, 386183668047020927, 2844591528, 1853821336, 281470681751456, 0, 0, 0, 0, 0, 0}}

        pid = <optimized out>

        tid = <optimized out>

        ret = <optimized out>

#1  0x00007f1ea8c8b801 in __GI_abort () at abort.c:79

        save_stage = 1

        act = {__sigaction_handler = {sa_handler = 0x3d2ef35793c76730, sa_sigaction = 0x3d2ef35793c76730}, sa_mask = {__val = {0, 0, 0, 4613425733568182272, 0, 139769661323011, 139769657786193, 11, 139769661322880, 10, 139768235051824, 139768679567824,

              139768235051792, 139768679567888, 139769657787395, 139769661322880}}, sa_flags = -1450548912, sa_restorer = 0x7f1e6a7740c8}

        sigs = {__val = {32, 0 <repeats 15 times>}}

        __cnt = <optimized out>

        __set = <optimized out>

        __cnt = <optimized out>

        __set = <optimized out>

#2  0x00007f1ea98a9165 in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95

        terminating = false

        t = <optimized out>

#3  0x000000000054cf72 in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:86

No locals.

#4  0x00007f1ea98a6f56 in __cxxabiv1::__terminate (handler=<optimized out>) at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47

No locals.

#5  0x00007f1ea98a6fa1 in std::terminate () at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57

No locals.

#6  0x00007f1ea98a71e3 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f1e54004b10, tinfo=0x7f1ea9b8aad8 <typeinfo for std::underflow_error>, dest=0x7f1ea98bc460 <std::underflow_error::~underflow_error()>)

    at /tmp/deploy/objdir/../gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:93

        globals = <optimized out>

        header = 0x7f1e54004a90

#7  0x00007f1eacfd0f14 in cb::ThrowExceptionUnderflowPolicy<unsigned long>::underflow (this=<optimized out>, desired=<optimized out>, current=<optimized out>, arg=<optimized out>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:48

No locals.

#8  0x00007f1ead1a4ade in cb::NonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::fetch_add (arg=-94, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:125

        current = <optimized out>

        desired = 18446744073709551522

        current = <optimized out>

        desired = <optimized out>

#9  cb::NonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::operator+= (rhs=18446744073709551522, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:182

No locals.

#10 Collections::VB::ManifestEntry::updateDiskSize (delta=-94, this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_entry.h:107

No locals.

#11 Collections::VB::Manifest::updateDiskSize (delta=-94, entry=..., this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest.h:1055

No locals.

#12 Collections::VB::Manifest::CachingReadHandle::updateDiskSize (delta=-94, this=0x7f1e6e7f1290) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest.h:393

No locals.

#13 Collections::VB::Flush::updateDiskSize (this=<optimized out>, key=..., delta=-94) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/flush.cc:58

No locals.

#14 0x00007f1ead16c3c2 in saveDocsCallback (oldInfo=0x7f1e6b12ce80, newInfo=0x7f1e6b12cef0, context=0x7f1e6e7f1a10) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:2113

        cbCtx = 0x7f1e6e7f1a10

        newKey = {keydata = {static npos = 18446744073709551615, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x7f1e6e7f1370 ""}, _M_string_length = 11, {

              _M_local_buf = "\000jepsen0004\000\000\000\000", _M_allocated_capacity = 3489838307740051968}}}

        onDiskMutationType = <optimized out>

        docKey = {<DocKeyInterface<DocKey>> = {<No data fields>}, buffer = {static npos = <optimized out>, buf = 0x7f1e6e7f1370 "", len = 11}, encoding = DocKeyEncodesCollectionId::Yes}

        oldSize = <optimized out>

        newSize = <optimized out>

        delta = <optimized out>

#15 0x00007f1eacd2733b in do_save_callback (valueNew=<optimized out>, valueOld=0x7f1e6e7f1440, key=0x7f1e6b01c540, rq=0x7f1e6e7f1610) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:431

        infoNew = 0x7f1e6b12cef0

        infoOld = 0x7f1e6b12ce80

        infoNew = <optimized out>

        infoOld = <optimized out>

#16 modify_node (rq=rq@entry=0x7f1e6e7f1610, nptr=nptr@entry=0x7f1e6b17c2c0, start=start@entry=0, end=1, dst=dst@entry=0x7f1e6a71d010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:537

        cmp_val = <optimized out>

        cmp_key = {buf = 0x7f1e6aeda146 <error: Cannot access memory at address 0x7f1e6aeda146>, size = 11}

        val_buf = {buf = 0x7f1e6aeda151 <error: Cannot access memory at address 0x7f1e6aeda151>, size = 48}

        advance = 1

        nodebuf = 0x7f1e6aeda140 <error: Cannot access memory at address 0x7f1e6aeda140>

        bufpos = 65

        nodebuflen = 130

        errcode = <optimized out>

        local_result = 0x7f1e6a71d0c8

#17 0x00007f1eacd27ca9 in modify_btree (rq=rq@entry=0x7f1e6e7f1610, root=0x7f1e6b17c2c0, errcode=errcode@entry=0x7f1e6e7f1544) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:809

        a = 0x7f1e6b06d400

        ret_ptr = 0x7f1e6b17c2c0

        root_result = <optimized out>

#18 0x00007f1eacd3bd70 in update_indexes (save_callback_ctx=<optimized out>, save_callback=<optimized out>, numdocs=<optimized out>, idvals=<optimized out>, ids=<optimized out>, seqvals=<optimized out>, seqs=0x7f1e6b01c530, db=<optimized out>, this=<optimized out>,

    this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:238

---Type <return> to continue, or q <return> to quit---

        size = 70

        seqrq = {cmp = {compare = 0x0}, file = 0x517d2e <cb_malloc(size_t)+14>, num_actions = 2, actions = 0x72, fetch_callback = 0x7f1e6e7f18a0, fetch_callback_ctx = 0x7f1ead57f8c9 <operator new(unsigned long)+9>, reduce = 0x200,

          rereduce = 0x42c4cc <std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long)+236>, user_reduce_ctx = 0x7f1e6e7f1928, purge_kp = 0x68, purge_kv = 0x7f1ea5650a80,

          enable_purging = 1, guided_purge_ctx = 0x7f1e6e7f1740, compacting = 1853823504, kv_chunk_threshold = 32542, kp_chunk_threshold = 1853823408, save_callback = 0xaaaaaaaaaaaaaaab, save_callback_ctx = 0x7f1e6e7f1760,

          docinfo_callback = 0x7f1ea8d7bea6 <__GI___clock_gettime+38>}

        errcode = COUCHSTORE_SUCCESS

        new_id_root = <optimized out>

        actbuf = {_M_t = {

            _M_t = {<std::_Tuple_impl<0, fatbuf*, cb::couchstore::FatbufDeletor>> = {<std::_Tuple_impl<1, cb::couchstore::FatbufDeletor>> = {<std::_Head_base<1, cb::couchstore::FatbufDeletor, true>> = {<cb::couchstore::FatbufDeletor> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, fatbuf*, false>> = {_M_head_impl = 0x7f1e6b07d9e0}, <No data fields>}, <No data fields>}}}

        seqacts = 0x7f1e6b07d9f0

        idacts = <optimized out>

        sorted_ids = <optimized out>

        fetcharg = {seqacts = 0x7f1e6b07d9f0, actpos = 1, valpos = 0, deltermbuf = @0x7f1e6b07d9e0}

        idrq = {cmp = {compare = 0x7f1eacd4a6a0 <ebin_cmp(sized_buf const*, sized_buf const*)>}, file = 0x7f1e6a7987a0, num_actions = 1, actions = 0x7f1e6b07da08,

          fetch_callback = 0x7f1eacd3b3d0 <idfetch_update_cb(couchfile_modify_request*, sized_buf*, sized_buf*, void*)>, fetch_callback_ctx = 0x7f1e6e7f1580, reduce = 0x7f1eacd49d30 <by_id_reduce(char*, size_t*, nodelist const*, int, void*)>,

          rereduce = 0x7f1eacd49e00 <by_id_rereduce(char*, size_t*, nodelist const*, int, void*)>, user_reduce_ctx = 0x0, purge_kp = 0x0, purge_kv = 0x0, enable_purging = 0, guided_purge_ctx = 0x0, compacting = 0, kv_chunk_threshold = 1279, kp_chunk_threshold = 1279,

          save_callback = 0x7f1ead16c2b0 <saveDocsCallback(DocInfo const*, DocInfo const*, void*)>, save_callback_ctx = 0x7f1e6e7f1a10, docinfo_callback = 0x7f1eacd2d790 <by_id_read_docinfo(_docinfo**, _sized_buf const*, _sized_buf const*)>}

        new_seq_root = <optimized out>

        size = <optimized out>

        actbuf = <optimized out>

        seqacts = <optimized out>

        idacts = <optimized out>

        sorted_ids = <optimized out>

        fetcharg = <optimized out>

        idrq = <optimized out>

        errcode = <optimized out>

        new_id_root = <optimized out>

        seqrq = <optimized out>

        new_seq_root = <optimized out>

        ii = <optimized out>

        ii = <optimized out>

        isorted = <optimized out>

#19 couchstore_save_documents_and_callback (db=<optimized out>, docs=<optimized out>, infos=<optimized out>, numdocs=<optimized out>, options=options@entry=3, save_cb=save_cb@entry=0x7f1ead16c2b0 <saveDocsCallback(DocInfo const*, DocInfo const*, void*)>,

    save_cb_ctx=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:424

        CL_item = {_M_t = {

            _M_t = {<std::_Tuple_impl<0, CouchLatencyItem*, std::default_delete<CouchLatencyItem> >> = {<std::_Tuple_impl<1, std::default_delete<CouchLatencyItem> >> = {<std::_Head_base<1, std::default_delete<CouchLatencyItem>, true>> = {<std::default_delete<CouchLatencyItem>> = {<No data fields>}, <No data fields>}, <No data fields>}, <std::_Head_base<0, CouchLatencyItem*, false>> = {_M_head_impl = 0x0}, <No data fields>}, <No data fields>}}}

        CL_item_ptr = {value = {_M_b = {

Segmentation fault (core dumped)

Looks like it might be related to http://review.couchbase.org/c/kv_engine/+/128356 as this code was only added a few days ago.

Attachments

Issue Links

duplicates

MB-39946 [Collections] Underflow of diskSize - total size (bytes) of items in this collection on disk

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: James Harrison (Inactive)

Reporter:: Richard deMellow

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 15/Jun/20 2:19 AM

Updated:: 08/Jul/20 2:20 PM

Resolved:: 15/Jun/20 2:26 AM

Gerrit Reviews

There are no open Gerrit changes

[Jepsen] Underflow of diskSize - total size (bytes) of items in this collection on disk

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty