Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.6.0, 7.2.4
Affects Version/s: 7.6.0, 7.2.1, 7.2.2, 7.2.3
Component/s: couchbase-bucket
Labels:
Environment:
7.6.0-1568

Triage:
Untriaged
Operating System:
Linux x86_64
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://cb-engineering.s3.amazonaws.com/gsi_test/test_2.tar.gz
Story Points:
0
Is this a Regression?:
Unknown
Sprint:
KV 2023-4

Description

Steps To Recreate:

Create a 4 node cluster
Create a magma bucket(replicas=1, bucket_name=test_bucket)
Create a scope (test_scope_1) and create a collection in newly create scope(test_collection_1)
Create 10K docs
Create below indexe
CREATE INDEX `idx1` ON default:test_bucket.test_scope_1.test_collection_1(age) USING GSI WITH {'defer_build': True}
Run below queries
select age, fistName from default:test_bucket.test_scope_1.test_collection_1 where age > 10 and age < 60 and firstName like "P%"
select age, country from default:test_bucket.test_scope_1.test_collection_1 where age > 10 and age < 60 and country like "S%"
select age, lastName from default:test_bucket.test_scope_1.test_collection_1 where age > 10 and age < 60 and lastName like "A%"
select age, city from default:test_bucket.test_scope_1.test_collection_1 where age > 10 and age < 60 and city like "C%"
Delete scope (test_scope_1)
Observed Memcached crashed at cb::AtomicNonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::load (this=<error reading variable: Cannot access memory at address 0x0>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:235

Below core was found on node 172.23.106.28 (a64483ba-7ff4-43bc-c33aaa9e-2d920d67.dmp). "172.23.106.28 OS" == debian 11

bt full:
https://gist.github.com/ankushsharma29/af51ac75687ca785f326fe84adcd8478

BackTrace:

(gdb) bt

#0  0x000000000086e3a4 in std::__atomic_base<unsigned long>::load (__m=std::memory_order_relaxed, this=<error reading variable: Cannot access memory at address 0x0>)

    at /opt/gcc-13.2.0/include/c++/13.2.0/bits/atomic_base.h:505

#1  cb::AtomicNonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::load (this=<error reading variable: Cannot access memory at address 0x0>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:235

#2  cb::AtomicNonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::operator unsigned long (

    this=<error reading variable: Cannot access memory at address 0x0>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:231

#3  Collections::VB::ManifestEntry::getItemCount (this=<error reading variable: Cannot access memory at address 0x0>)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_entry.h:135

#4  Collections::VB::StatsReadHandle::getItemCount (this=this@entry=0x7fd27b7f3070)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_handles.cc:43

#5  0x00000000009bd84b in ActiveStream::tryAndScheduleOSOBackfill (this=0x7fd1e9252f90, producer=..., vb=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:2068

#6  0x00000000009c1385 in ActiveStream::scheduleBackfill_UNLOCKED (this=0x7fd1e9252f90, producer=..., reschedule=<optimized out>)

    at /opt/gcc-13.2.0/include/c++/13.2.0/bits/shared_ptr_base.h:1349

#7  0x00000000009bf812 in ActiveStream::transitionState (this=0x7fd1e9252f90, newState=ActiveStream::StreamState::Backfilling)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:2408

#8  0x00000000009caf0d in ActiveStream::setActive (this=0x7fd1e9252f90)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.h:138

#9  0x00000000008b3b2e in DcpProducer::streamRequest(unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (this=<optimized out>, flags=<optimized out>, opaque=<optimized out>, vbucket=..., start_seqno=<optimized out>,

    end_seqno=18446744073709551615, vbucket_uuid=0, snap_start_seqno=0, snap_end_seqno=0, rollback_seqno=<optimized out>, callback=..., json=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/producer.cc:612

#10 0x0000000000737457 in EventuallyPersistentEngine::stream_req(CookieIface&, unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (this=<optimized out>, cookie=..., flags=16, opaque=1275200256, vbucket=..., startSeqno=0,

    endSeqno=18446744073709551615, vbucketUuid=0, snapStartSeqno=0, snapEndSeqno=0, rollbackSeqno=0x7fd27b7f3ad8, callback=..., json=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:1263

#11 0x00000000005d28a6 in dcpStreamReq(Cookie&, unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (cookie=..., flags=flags@entry=16, opaque=1275200256, vbucket=..., vbucket@entry=..., startSeqno=startSeqno@entry=0,

    endSeqno=endSeqno@entry=18446744073709551615, vbucketUuid=0, snapStartSeqno=0, snapEndSeqno=0, rollbackSeqno=0x7fd27b7f3ad8, callback=..., json=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:898

#12 0x000000000069eca8 in dcp_stream_req_executor (cookie=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/dcp_stream_req_executor.cc:59

#13 0x000000000061e765 in Cookie::doExecute (this=0x7fd284cec100)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/mcbp/protocol/header.h:110

#14 0x000000000061e809 in Cookie::execute (this=this@entry=0x7fd284cec100, useStartTime=useStartTime@entry=true)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:189

#15 0x0000000000605270 in Connection::executeCommandPipeline (this=0x7fd234785100)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:665

#16 0x000000000060a194 in Connection::executeCommandsCallback (this=this@entry=0x7fd234785100)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:985

--Type <RET> for more, q to quit, c to continue without paging--

#17 0x0000000000614397 in LibeventConnection::read_callback (this=0x7fd234785100)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection_libevent.cc:135

#18 0x00007fd2870d184e in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7fd234785380)

    at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208

#19 0x00007fd2870dab99 in event_process_active_single_queue (base=base@entry=0x7fd284cd8200, activeq=0x7fd284d6c080, max_to_process=max_to_process@entry=2147483647,

    endtime=endtime@entry=0x0) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726

#20 0x00007fd2870db287 in event_process_active (base=0x7fd284cd8200)

    at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789

#21 event_base_loop (base=0x7fd284cd8200, flags=flags@entry=1)

    at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012

#22 0x0000000000d7d5c6 in (anonymous namespace)::EventBaseBackend::eb_event_base_loop (flags=1, this=<optimized out>)

    at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:74

#23 folly::EventBase::loopBody (this=0x7fd284dc9c80, flags=<optimized out>, ignoreKeepAlive=<optimized out>)

    at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:381

#24 0x0000000000d7dace in folly::EventBase::loop (this=0x7fd284dc9c80)

    at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:305

#25 0x0000000000d7ff98 in folly::EventBase::loopForever (this=this@entry=0x7fd284dc9c80)

    at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:524

#26 0x00000000005c4829 in worker_libevent (arg=0x7fd284dc9c70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:272

#27 0x0000000000cf0d9c in std::function<void ()>::operator()() const (this=0x7fd284c97418) at /opt/gcc-13.2.0/include/c++/13.2.0/bits/std_function.h:591

#28 operator() (__closure=0x7fd284c973f8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:117

#29 std::__invoke_impl<void, create_thread(std::function<void()>, std::string)::<lambda()> > (__f=...) at /opt/gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:61

#30 std::__invoke<create_thread(std::function<void()>, std::string)::<lambda()> > (__fn=...) at /opt/gcc-13.2.0/include/c++/13.2.0/bits/invoke.h:96

#31 std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > >::_M_invoke<0> (this=0x7fd284c973f8)

    at /opt/gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:292

#32 std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > >::operator() (this=0x7fd284c973f8)

    at /opt/gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:299

#33 std::thread::_State_impl<std::thread::_Invoker<std::tuple<create_thread(std::function<void()>, std::string)::<lambda()> > > >::_M_run(void) (this=0x7fd284c973f0)

    at /opt/gcc-13.2.0/include/c++/13.2.0/bits/std_thread.h:244

#34 0x00007fd2863f1aa3 in std::execute_native_thread_routine (__p=0x7fd284c973f0) at /tmp/deploy/objdir/../gcc-13.2.0/libstdc++-v3/src/c++11/thread.cc:104

#35 0x00007fd286a71ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477

#36 0x00007fd2860c9a2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

QE-TEST:

./testrunner -i /data/workspace/debian-p0-2i-vset00-00-collections_gsi_delete_bsc_7.0_P1/testexec.27242.ini -p gsi_type=plasma,get-cbcollect-info=True,reset_services=True,java_sdk_client=True,GROUP=P1,get-cbcollect-info=True,sirius_url=http://172.23.120.103:4000 -t gsi.collections_index_delete_bsc.CollectionsIndexDeleteBSC.test_index_during_scan_with_delete_bsc,default_bucket=false,force_clean=true,defer_build=True,skip_load=True,item_to_delete=scope,GROUP=P1

Job: http://qa.sc.couchbase.com/job/test_suite_executor/623220/consoleFull
Job_Name: collections_gsi_delete_bsc_7.0_P1(component: 2i)

Attachments

Issue Links

is duplicated by

MB-59096 memcached crashes in Collections::VB::ManifestEntry::getItemCount

Closed

MB-59472 Memcached crashed at cb::AtomicNonNegativeCounter<unsigned long, cb::ClampAtZeroUnderflowPolicy>::load (this=<error reading variable: Cannot access memory at address 0x0>)