Details
-
Bug
-
Resolution: Fixed
-
Major
-
master
-
None
-
Untriaged
-
Unknown
Description
RocksDBKVStore keeps a vector of unique pointers to opened DBs.
It is possible that a opened DB is deleted in RocksDBKVStore::delVBucket() while the DB is live (i.e., the DB "dangling" pointer is being used somewhere else). E.g.:
(gdb) bt
|
#0 rocksdb::DBImpl::GetAggregatedIntProperty (this=0x7f03141df400, property=..., aggregated_value=0x7f0465ded2d8) at db/db_impl.cc:1773
|
#1 0x00007f04760a20e6 in rocksdb::MemoryUtil::GetApproximateMemoryUsageByType (dbs=std::vector of length 127, capacity 128 = {...}, cache_set=std::unordered_set with 2 elements = {...},
|
usage_by_type=usage_by_type@entry=0x7f0465ded3e0) at utilities/memory/memory_util.cc:23
|
#2 0x00007f047676aca2 in RocksDBKVStore::getStat (this=0x7f025cc2a400, name=0x7f047678e3c2 "io_total_write_bytes", value=@0x7f0465ded490: 139657227933936)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:620
|
#3 0x00007f04766ed9de in KVBucket::getKVStoreStat (this=<optimized out>, name=0x7f047678e3c2 "io_total_write_bytes", value=@0x7f0465ded500: 837943367603616, option=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kv_bucket.cc:2178
|
#4 0x00007f04766a7bc9 in EventuallyPersistentEngine::doEngineStats (this=0x7f045476a000, cookie=0x7f0454356b78, add_stat=0x471b30 <append_stats(char const*, uint16_t, char const*, uint32_t, void const*)>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:2630
|
#5 0x00007f04766af400 in EventuallyPersistentEngine::getStats (this=this@entry=0x7f045476a000, cookie=cookie@entry=0x7f0454356b78, stat_key=stat_key@entry=0x0, nkey=nkey@entry=0,
|
add_stat=add_stat@entry=0x471b30 <append_stats(char const*, uint16_t, char const*, uint32_t, void const*)>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:3586
|
#6 0x00007f04766b0288 in EvpGetStats (handle=0x7f045476a000, cookie=0x7f0454356b78, stat_key=0x0, nkey=0, add_stat=0x471b30 <append_stats(char const*, uint16_t, char const*, uint32_t, void const*)>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:316
|
#7 0x0000000000476f8a in get_stats (k=..., this=0x7f0454736050) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:924
|
#8 StatsCommandContext::step (this=0x7f0454736050) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/stats_context.cc:850
|
#9 0x0000000000479a84 in SteppableCommandContext::drive (this=this@entry=0x7f0454736050) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/protocol/mcbp/steppable_command_context.cc:33
|
#10 0x0000000000455655 in stat_executor (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:233
|
#11 0x0000000000457511 in operator() (__args#0=..., this=0x18bf600 <handlers+512>) at /usr/include/c++/4.8.2/functional:2471
|
#12 execute_request_packet (request=..., cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:844
|
#13 mcbp_execute_packet (cookie=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/mcbp_executors.cc:925
|
#14 0x000000000047c214 in conn_execute (connection=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/statemachine_mcbp.cc:372
|
#15 0x000000000047d53e in McbpStateMachine::execute (this=this@entry=0x7f0454356998) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/statemachine_mcbp.cc:170
|
#16 0x000000000044526c in McbpConnection::runStateMachinery (this=0x7f0454356880) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection_mcbp.cc:783
|
#17 0x00000000004452ab in McbpConnection::runEventLoop (this=0x7f0454356880, which=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connection_mcbp.cc:952
|
#18 0x0000000000447ffe in run_event_loop (c=0x7f0454356880, which=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/connections.cc:159
|
#19 0x0000000000418dbc in event_handler (fd=23743, which=<optimized out>, arg=0x7f0454356880) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1040
|
#20 0x00007f047c23542c in event_process_active_single_queue.isra.26 () from /opt/couchbase/bin/../lib/libevent_core.so.2.1.8
|
#21 0x00007f047c23887f in event_base_loop () from /opt/couchbase/bin/../lib/libevent_core.so.2.1.8
|
#22 0x0000000000432804 in worker_libevent (arg=0x7f046e38ec80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:232
|
#23 0x00007f047d11802e in run (this=0x7f046e29ece0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:59
|
#24 platform_thread_wrap (arg=0x7f046e29ece0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:72
|
#25 0x00007f047cadbdc5 in start_thread () from /lib64/libpthread.so.0
|
#26 0x00007f047ac7876d in clone () from /lib64/libc.so.6
|
(gdb) t 130
|
[Switching to thread 130 (Thread 0x7f02535f4700 (LWP 14207))]
|
#0 0x00007f047cae21bd in __lll_lock_wait () from /lib64/libpthread.so.0
|
(gdb) bt
|
#0 0x00007f047cae21bd in __lll_lock_wait () from /lib64/libpthread.so.0
|
#1 0x00007f047caddd02 in _L_lock_791 () from /lib64/libpthread.so.0
|
#2 0x00007f047caddc08 in pthread_mutex_lock () from /lib64/libpthread.so.0
|
#3 0x00007f0475fdbda9 in rocksdb::port::Mutex::Lock (this=this@entry=0x7f03141df830) at port/port_posix.cc:60
|
#4 0x00007f0475fb7255 in rocksdb::InstrumentedMutex::LockInternal (this=this@entry=0x7f03141df830) at monitoring/instrumented_mutex.cc:38
|
#5 0x00007f0475fb7346 in rocksdb::InstrumentedMutex::Lock (this=0x7f03141df830) at monitoring/instrumented_mutex.cc:26
|
#6 0x00007f0475e9d051 in rocksdb::ColumnFamilyData::~ColumnFamilyData (this=0x7f025c4c6400, __in_chrg=<optimized out>) at db/column_family.cc:453
|
#7 0x00007f0475e9df28 in rocksdb::ColumnFamilySet::~ColumnFamilySet (this=0x7f025cd76da0, __in_chrg=<optimized out>) at db/column_family.cc:1002
|
#8 0x00007f0475f6914b in operator() (this=<optimized out>, __ptr=0x7f025cd76da0) at /usr/include/c++/4.8.2/bits/unique_ptr.h:67
|
#9 reset (__p=0x7f025cd76da0, this=0x7f025c5bf940) at /usr/include/c++/4.8.2/bits/unique_ptr.h:262
|
#10 rocksdb::VersionSet::~VersionSet (this=0x7f025c5bf940, __in_chrg=<optimized out>) at db/version_set.cc:2340
|
#11 0x00007f0475edd8f3 in operator() (this=<optimized out>, __ptr=0x7f025c5bf940) at /usr/include/c++/4.8.2/bits/unique_ptr.h:67
|
#12 reset (__p=0x7f025c5bf940, this=0x7f03141df418) at /usr/include/c++/4.8.2/bits/unique_ptr.h:262
|
#13 rocksdb::DBImpl::~DBImpl (this=0x7f03141df400, __in_chrg=<optimized out>) at db/db_impl.cc:349
|
#14 0x00007f0475eddf01 in rocksdb::DBImpl::~DBImpl (this=0x7f03141df400, __in_chrg=<optimized out>) at db/db_impl.cc:357
|
#15 0x00007f04767674b0 in operator() (this=<optimized out>, __ptr=<optimized out>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:67
|
#16 ~unique_ptr (this=0x7f025c538540, __in_chrg=<optimized out>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:184
|
#17 ~KVRocksDB (this=0x7f025c538540, __in_chrg=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:164
|
#18 operator() (this=<optimized out>, __ptr=0x7f025c538540) at /usr/include/c++/4.8.2/bits/unique_ptr.h:67
|
#19 reset (__p=0x7f025c538540, this=<optimized out>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:262
|
#20 RocksDBKVStore::delVBucket (this=0x7f025cc2a400, vbid=<optimized out>, vb_version=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/rocksdb-kvstore/rocksdb-kvstore.cc:534
|
#21 0x00007f047672f79a in VBucketMemoryAndDiskDeletionTask::run (this=0x7f02660d25b8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucketdeletiontask.cc:87
|
#22 0x00007f04766cddc1 in ExecutorThread::run (this=0x7f0259592c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:122
|
#23 0x00007f047d11802e in run (this=0x7f02595ec120) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:59
|
#24 platform_thread_wrap (arg=0x7f02595ec120) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:72
|
#25 0x00007f047cadbdc5 in start_thread () from /lib64/libpthread.so.0
|
#26 0x00007f047ac7876d in clone () from /lib64/libc.so.6
|
Attachments
For Gerrit Dashboard: MB-26849 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
85650,13 | MB-26849 [RocksDB] Avoid data races caused by 'delVBucket' | master | kv_engine | Status: MERGED | +2 | +1 |