Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
7.2.3
-
7.2.3-6705
-
Untriaged
-
Linux x86_64
-
0
-
Unknown
Description
Steps to Recreate:
Steps:
- Create a 6 node cluster (3 kv nodes, 2 index , 1 index:n1ql)
- Create 1 couchstore, 1 ephemeral , 1 magma buckets)
- Set oso_dcp_backfill
- Create few indexes and run few queries
- SIG Kill memcached on all the kv nodes one by one
This issue looks similar to MB-58893
Core was found on node 172.23.106.100(1fb73785-a4e5-4bec-b6d8a5a0-73ad28b9.dmp)
bt full:
https://gist.github.com/ankushsharma29/fb9ee32f6fb40330e7226742e0d1bef4
BackTrace:
(gdb) bt
|
#0 0x00000000007d5a74 in std::__atomic_base<unsigned long>::load (__m=std::memory_order_relaxed, this=<error reading variable: Cannot access memory at address 0x0>)
|
at /opt/gcc-10.2.0/include/c++/10.2.0/bits/atomic_base.h:426
|
#1 cb::AtomicNonNegativeCounter<unsigned long, cb::ClampAtZeroUnderflowPolicy>::load (this=<error reading variable: Cannot access memory at address 0x0>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:235
|
#2 cb::AtomicNonNegativeCounter<unsigned long, cb::ClampAtZeroUnderflowPolicy>::operator unsigned long (
|
this=<error reading variable: Cannot access memory at address 0x0>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/non_negative_counter.h:231
|
#3 Collections::VB::ManifestEntry::getItemCount (this=<error reading variable: Cannot access memory at address 0x0>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_entry.h:110
|
#4 Collections::VB::StatsReadHandle::getItemCount (this=this@entry=0x7f55df7eb450)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/collections/vbucket_manifest_handles.cc:39
|
#5 0x00000000008e2e7d in operator() (__closure=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1915
|
#6 ActiveStream::tryAndScheduleOSOBackfill (this=0x7f5556486090, producer=..., vb=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:1916
|
#7 0x00000000008e572c in ActiveStream::scheduleBackfill_UNLOCKED (this=<optimized out>, producer=..., reschedule=<optimized out>)
|
at /opt/gcc-10.2.0/include/c++/10.2.0/bits/shared_ptr_base.h:1324
|
#8 0x00000000008e6452 in ActiveStream::transitionState (this=0x7f5556486090, newState=ActiveStream::StreamState::Backfilling)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.cc:2238
|
#9 0x00000000008f1b1d in ActiveStream::setActive (this=0x7f5556486090)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/active_stream.h:138
|
#10 0x000000000080c5a3 in DcpProducer::streamRequest(unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (this=this@entry=0x7f5560ae4280, flags=flags@entry=16, opaque=opaque@entry=2768372744, vbucket=vbucket@entry=...,
|
start_seqno=start_seqno@entry=0, end_seqno=end_seqno@entry=18446744073709551615, vbucket_uuid=<optimized out>, snap_start_seqno=<optimized out>,
|
snap_end_seqno=<optimized out>, rollback_seqno=<optimized out>, callback=..., json=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/producer.cc:577
|
#11 0x00000000006bacd6 in EventuallyPersistentEngine::stream_req(CookieIface const&, unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (this=<optimized out>, cookie=..., flags=16, opaque=2768372744, vbucket=..., startSeqno=0,
|
endSeqno=18446744073709551615, vbucketUuid=0, snapStartSeqno=0, snapEndSeqno=0, rollbackSeqno=0x7f55df7ec028, callback=..., json=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:1310
|
|
#12 0x000000000059e65b in dcpStreamReq(Cookie&, unsigned int, unsigned int, Vbid, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long*, std::function<cb::engine_errc (std::vector<vbucket_failover_t, std::allocator<vbucket_failover_t> > const&)>, std::optional<std::basic_string_view<char, std::char_traits<char> > >) (cookie=..., flags=flags@entry=16, opaque=2768372744, vbucket=vbucket@entry=..., startSeqno=startSeqno@entry=0,
|
endSeqno=endSeqno@entry=18446744073709551615, vbucketUuid=0, snapStartSeqno=0, snapEndSeqno=0, rollbackSeqno=0x7f55df7ec028, callback=..., json=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/engine_wrapper.cc:724
|
#13 0x00000000006411d8 in dcp_stream_req_executor (cookie=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/dcp_stream_req_executor.cc:59
|
#14 0x000000000061361b in Cookie::doExecute (this=0x7f55eccd0080)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/mcbp/protocol/header.h:110
|
#15 0x00000000006136b0 in Cookie::execute (this=this@entry=0x7f55eccd0080, useStartTime=useStartTime@entry=true)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/cookie.cc:168
|
#16 0x00000000005f4bd3 in Connection::executeCommandPipeline (this=0x7f55f0c4d580)
|
--Type <RET> for more, q to quit, c to continue without paging--
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:577
|
#17 0x00000000005f4e82 in Connection::executeCommandsCallback (this=0x7f55f0c4d580)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:801
|
#18 0x00000000005f550b in Connection::rw_callback (bev=<optimized out>, ctx=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection.cc:950
|
#19 0x00007f55f31efd7e in bufferevent_run_deferred_callbacks_unlocked (cb=<optimized out>, arg=0x7f55ecdde000)
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/bufferevent.c:208
|
#20 0x00007f55f31f8cf1 in event_process_active_single_queue (base=base@entry=0x7f55f0c66c00, max_to_process=max_to_process@entry=2147483647,
|
endtime=endtime@entry=0x0, activeq=<optimized out>) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1726
|
#21 0x00007f55f31f953f in event_process_active (base=0x7f55f0c66c00)
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1789
|
#22 event_base_loop (base=0x7f55f0c66c00, flags=1) at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:2012
|
#23 0x0000000000c2f3f7 in folly::EventBase::loopBody (this=this@entry=0x7f55f0c3ac40, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false)
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397
|
#24 0x0000000000c2f8c6 in folly::EventBase::loop (this=this@entry=0x7f55f0c3ac40)
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315
|
#25 0x0000000000c31286 in folly::EventBase::loopForever (this=this@entry=0x7f55f0c3ac40)
|
at /home/couchbase/jenkins/cbdeps-ws/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538
|
#26 0x00000000005c16f9 in worker_libevent (arg=0x7f55f0c3ac30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:257
|
#27 0x0000000000bb3ed9 in CouchbaseThread::run (this=0x7f55f0c220a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51
|
#28 platform_thread_wrap (arg=0x7f55f0c220a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64
|
#29 0x00007f55f3d28ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
|
#30 0x00007f55f1f8aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
QE-TEST:
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-collections-vset00-00-dcp_oso_backfill_7.2_P1_majority/testexec.11104.ini GROUP=OSO_DCP_BACKFILL_P0,get-cbcollect-info=True,infra_log_level=info,log_level=info,bucket_storage=couchstore,upgrade_version=7.2.3-6705,sirius_url=http://172.23.120.103:4000 -t bucket_collections.oso_backfill_test.KvOsoBackfillTests.test_dcp_backfill_config,nodes_init=6,replicas=1,num_items=5000,load_collections_exponentially=True,services_init=kv-kv-kv-index-index-index:n1ql,bucket_spec=multi_bucket.buckets_with_similar_hierarchy,oso_dcp_backfill=auto,simulate_error=kill_memcached,kv_quota_percent=100,index_quota_percent=100,err_itrs=20,GROUP=OSO_DCP_BACKFILL_P0'
|
Job: http://qa.sc.couchbase.com/job/test_suite_executor-TAF/282453/consoleFull
Attachments
Issue Links
- duplicates
-
MB-58893 Memcached crashed at cb::AtomicNonNegativeCounter<unsigned long, cb::ThrowExceptionUnderflowPolicy>::load (this=<error reading variable: Cannot access memory at address 0x0>)
- Closed