Memcached crashed in CouchKVStore::readVBState(_db*, Vbid) () from /opt/couchbase/bin/../lib/libep.so

172.23.122.110 : Stack Trace of first crash: f0470eed-54b9-42fd-3d110eae-9a1e2bd3.dmp

Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.

 #0  0x00007fac55dad337 in raise () from /lib64/libc.so.6

 #0  0x00007fac55dad337 in raise () from /lib64/libc.so.6

 #1  0x00007fac55daea28 in abort () from /lib64/libc.so.6

 #2  0x00007fac5690b195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6

 #3  0x00000000005a4bd2 in backtrace_terminate_handler() ()

 #4  0x00007fac56908f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6

 #5  0x00007fac56908fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6

 #6  0x00007fac56909213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6

 #7  0x00007fac56931867 in std::__throw_out_of_range(char const*) () from /opt/couchbase/bin/../lib/libstdc++.so.6

 #8  0x00007fac5a01e987 in from_json(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> const&, vbucket_state&) () from /opt/couchbase/bin/../lib/libep.so

 #9  0x00007fac5a064394 in CouchKVStore::readVBState(_db*, Vbid) () from /opt/couchbase/bin/../lib/libep.so

 #10 0x00007fac5a066fc1 in CouchKVStore::initBySeqnoScanContext(std::unique_ptr<StatusCallback<GetValue>, std::default_delete<StatusCallback<GetValue> > >, std::unique_ptr<StatusCallback<CacheLookup>, std::default_delete<StatusCallback<CacheLookup> > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource) () from /opt/couchbase/bin/../lib/libep.so

 #11 0x00007fac59ee5d58 in DCPBackfillBySeqnoDisk::create() () from /opt/couchbase/bin/../lib/libep.so

 #12 0x00007fac59ee9479 in DCPBackfillDisk::run() () from /opt/couchbase/bin/../lib/libep.so

 #13 0x00007fac59eeb4eb in BackfillManager::backfill() () from /opt/couchbase/bin/../lib/libep.so

 #14 0x00007fac59eeb74e in BackfillManagerTask::run() () from /opt/couchbase/bin/../lib/libep.so

 #15 0x00007fac59faba03 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so

 #16 0x00007fac59eaf65f in CB3ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so

 #17 0x00007fac58907947 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0

 #18 0x00007fac5614ce65 in start_thread () from /lib64/libpthread.so.0

 #19 0x00007fac55e7588d in clone () from /lib64/libc.so.6

/data/workspace/centos-p0-transaction-vset00-00-rebalance_in_out_durability_majority_6.5_P0/job_logs/testrunner-20-Sep-22_21-18-47/test_2/collectinfo-2020-09-23T045129-ns_1@172.23.122.110.zip

cbcollect_info_ns_1@172.23.122.110_20200923-045129/f0470eed-54b9-42fd-3d110eae-9a1e2bd3.dmp

2020-09-22T21:45:21.308302-07:00 CRITICAL *** Fatal error encountered during exception handling ***

2020-09-22T21:45:21.308348-07:00 CRITICAL Caught unhandled std::exception-derived exception. what(): stoll

2020-09-22T21:45:21.474821-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-3180). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/f0470eed-54b9-42fd-3d110eae-9a1e2bd3.dmp before terminating.

2020-09-22T21:45:21.474846-07:00 CRITICAL Stack backtrace of crashed thread:

2020-09-22T21:45:21.475041-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x193e8d]

2020-09-22T21:45:21.475057-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x1aa20a]

2020-09-22T21:45:21.475070-07:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x1aa548]

2020-09-22T21:45:21.475077-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7fac56145000+0xf5f0]

2020-09-22T21:45:21.475099-07:00 CRITICAL     /lib64/libc.so.6(gsignal+0x37) [0x7fac55d77000+0x36337]

2020-09-22T21:45:21.475118-07:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7fac55d77000+0x37a28]

2020-09-22T21:45:21.475145-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x125) [0x7fac5687a000+0x91195]

2020-09-22T21:45:21.475158-07:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x1a4bd2]

2020-09-22T21:45:21.475173-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fac5687a000+0x8ef86]

2020-09-22T21:45:21.475188-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fac5687a000+0x8efd1]

2020-09-22T21:45:21.475202-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fac5687a000+0x8f213]

2020-09-22T21:45:21.475227-07:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt20__throw_out_of_rangePKc+0x37) [0x7fac5687a000+0xb7867]

2020-09-22T21:45:21.475238-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0x1f1987]

2020-09-22T21:45:21.475247-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0x237394]

2020-09-22T21:45:21.475257-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0x239fc1]

2020-09-22T21:45:21.475265-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0xb8d58]

2020-09-22T21:45:21.475272-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0xbc479]

2020-09-22T21:45:21.475278-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0xbe4eb]

2020-09-22T21:45:21.475284-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0xbe74e]

2020-09-22T21:45:21.475292-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0x17ea03]

2020-09-22T21:45:21.475299-07:00 CRITICAL     /opt/couchbase/bin/../lib/libep.so() [0x7fac59e2d000+0x8265f]

2020-09-22T21:45:21.475305-07:00 CRITICAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fac588f7000+0x10947]

2020-09-22T21:45:21.475311-07:00 CRITICAL     /lib64/libpthread.so.0() [0x7fac56145000+0x7e65]

2020-09-22T21:45:21.475340-07:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7fac55d77000+0xfe88d]

rebalance_new.rebalance_in_out.RebalanceInOutTests.test_incremental_rebalance_in_out_with_mutation_and_compaction,nodes_init=3,doc_size=512,transaction_timeout=200,num_items=10000,atomicity=True,durability=MAJORITY,bucket_storage=couchstore

git clone https://github.com/couchbaselabs/TAF.git

cd TAF

git clone https://github.com/gradle/guides.git

./guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.0 -P client_version=3.0.8 -P 'args=-i nodes.ini -t rebalance_new.rebalance_in_out.RebalanceInOutTests.test_incremental_rebalance_in_out_with_mutation_and_compaction,nodes_init=3,doc_size=512,transaction_timeout=200,num_items=10000,atomicity=True,durability=MAJORITY,bucket_storage=couchstore -m rest'