Details
-
Bug
-
Resolution: Unresolved
-
Critical
-
6.5.0
-
None
-
Enterprise Edition 6.5.0 build 1430 ‧ IPv4
-
Untriaged
-
-
Unknown
Description
Script to Repro
./testrunner -i /tmp/win10-bucket-ops.ini -p -t rebalance.rebalance_high_ops_pillowfight.RebalanceHighOpsWithPillowFight.test_rebalance_in_out,nodes_in=1,nodes_out=2,replicas=2,nodes_init=4,items=2000000,batch_size=1000,rate_limit=100000,instances=2,threads=5,loader=high_ops
|
Steps:
1) Create a 4 node cluster
2) Start data loading
3) When 2) is in progess, add 1 node and rebalance out 2 nodes.
4) Data loss is observed
Expected Docs : 6,000,00 Actual Docs: 5,966,696
Saw the following message in the logs (on 172.23.120.201)
Service 'memcached' exited with status 139. Restarting. Messages:
|
2018-10-15T04:28:48.244654Z WARNING (default) getNumItemsForCursor_UNLOCKED(): Cursor not found in the checkpoint manager on vb:655
|
2018-10-15T04:28:49.237078Z WARNING 81: Slow operation. {"cid":"172.23.107.68:55724/ed30000","duration":"924 ms","trace":"request=2907622155870916:924296","command":"SET","peer":"172.23.107.68:55724"}
|
2018-10-15T04:28:49.237304Z WARNING 56: Slow operation. {"cid":"172.23.107.68:55632/d5e20000","duration":"922 ms","trace":"request=2907622158458936:922110","command":"SET","peer":"172.23.107.68:55632"}
|
2018-10-15T04:28:49.238944Z WARNING 72: Slow operation. {"cid":"172.23.107.68:55678/25db0000","duration":"915 ms","trace":"request=2907622166832134:915340","command":"SET","peer":"172.23.107.68:55678"}
|
2018-10-15T04:28:49.241937Z WARNING (default) Slow runtime for 'Checkpoint Remover on vb:334' on thread nonIO_worker_2: 973 ms
|
2018-10-15T04:28:49.281832Z CRITICAL Breakpad caught a crash (Couchbase version 6.5.0-1430). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/2d70a881-298e-5f65-444c802d-57fed04a.dmp before terminating
|
Backtrace from the coredump(172.23.120.201)
(gdb) bt
|
#0 _M_find_before_node (__code=352256, __k=<optimized out>, __n=16, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/hashtable.h:1545
|
#1 _M_find_node (__c=352256, __key=<optimized out>, __bkt=16, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/hashtable.h:642
|
#2 find (__k=<optimized out>, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/hashtable.h:1422
|
#3 find (__x=<optimized out>, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/unordered_map.h:923
|
#4 findBuffer (offset=352256, h=0x7f071d948000, this=0x7f07d1a784c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/iobuffer.cc:122
|
#5 BufferedFileOps::pread (errinfo=0x7f07d1a9d120, handle=0x7f071d948000, buf=0x7f0806f34e24, nbyte=190, offset=352257, this=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/iobuffer.cc:402
|
#6 0x00007f07dbb59393 in read_skipping_prefixes (dst=0x7f0806f34e24, len=190, pos=<synthetic pointer>, file=0x7f07d1a9d100)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_file_read.cc:101
|
#7 pread_bin_internal (file=0x7f07d1a9d100, pos=352257, ret_ptr=ret_ptr@entry=0x7f07a97f8838, max_header_size=max_header_size@entry=0)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_file_read.cc:150
|
#8 0x00007f07dbb59774 in pread_compressed (file=<optimized out>, pos=<optimized out>, ret_ptr=ret_ptr@entry=0x7f07a97f88d8)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_file_read.cc:181
|
#9 0x00007f07dbb4cb47 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071e062f2c, start=start@entry=1474, end=end@entry=1476, dst=dst@entry=0x7f071e060ab4)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:445
|
#10 0x00007f07dbb4ccc5 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071e060a84, start=1474, start@entry=1404, end=end@entry=1506, dst=dst@entry=0x7f071e2cc1b0)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:610
|
#11 0x00007f07dbb4ccc5 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071e2cc180, start=1404, start@entry=0, end=end@entry=2186, dst=dst@entry=0x7f071e2cc0c8)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:610
|
#12 0x00007f07dbb4ccc5 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071d9480c0, start=start@entry=0, end=3146, dst=dst@entry=0x7f071e2cc010)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:610
|
#13 0x00007f07dbb4db90 in modify_btree (rq=rq@entry=0x7f07a97f8dc0, root=0x7f071d9480c0, errcode=errcode@entry=0x7f07a97f8c74)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:768
|
#14 0x00007f07dbb5b6cc in update_indexes (save_callback_ctx=0x7f07a97f93e0, save_callback=0x7f07dbeec600 <saveDocsCallback(DocInfo const*, DocInfo const*, void*)>, numdocs=1573, idvals=<optimized out>,
|
ids=0x7f071bdb8260, seqvals=<optimized out>, seqs=<optimized out>, db=0x7f07d1a9d100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:238
|
#15 couchstore_save_documents_and_callback (db=<optimized out>, docs=<optimized out>, infos=<optimized out>, numdocs=<optimized out>, options=options@entry=3,
|
save_cb=save_cb@entry=0x7f07dbeec600 <saveDocsCallback(DocInfo const*, DocInfo const*, void*)>, save_cb_ctx=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:407
|
#16 0x00007f07dbef5140 in CouchKVStore::saveDocs (this=this@entry=0x7f07d1b91800, vbid=..., docs=..., docinfos=..., kvctx=..., collectionsFlush=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:2039
|
#17 0x00007f07dbef5d6d in CouchKVStore::commit2couchstore (this=0x7f07d1b91800, collectionsFlush=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:1942
|
#18 0x00007f07dbef6685 in CouchKVStore::commit (this=0x7f07d1b91800, collectionsFlush=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:1209
|
#19 0x00007f07dbe21a29 in EPBucket::commit (this=0x7f07d1b5c300, kvstore=..., collectionsFlush=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:500
|
#20 0x00007f07dbe2276d in EPBucket::flushVBucket (this=0x7f07d1b5c300, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:434
|
#21 0x00007f07dbe66ab5 in Flusher::flushVB (this=0x7f07d1a6dcc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:300
|
#22 0x00007f07dbe66fdd in Flusher::step (this=0x7f07d1a6dcc0, task=0x7f07d1a733b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:207
|
#23 0x00007f07dbe5deb8 in ExecutorThread::run (this=0x7f07d1a6d400) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:148
|
#24 0x00007f080b0fc0e7 in run (this=0x7f07d7dbe3d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:59
|
#25 platform_thread_wrap (arg=0x7f07d7dbe3d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:72
|
#26 0x00007f08092ecdc5 in start_thread () from /usr/lib64/libpthread-2.17.so
|
#27 0x00007f080901a21d in clone () from /usr/lib64/libc-2.17.so
|
(gdb)
|
#0 _M_find_before_node (__code=352256, __k=<optimized out>, __n=16, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/hashtable.h:1545
|
#1 _M_find_node (__c=352256, __key=<optimized out>, __bkt=16, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/hashtable.h:642
|
#2 find (__k=<optimized out>, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/hashtable.h:1422
|
#3 find (__x=<optimized out>, this=0x7f07d1a784d8) at /usr/local/include/c++/7.3.0/bits/unordered_map.h:923
|
#4 findBuffer (offset=352256, h=0x7f071d948000, this=0x7f07d1a784c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/iobuffer.cc:122
|
#5 BufferedFileOps::pread (errinfo=0x7f07d1a9d120, handle=0x7f071d948000, buf=0x7f0806f34e24, nbyte=190, offset=352257, this=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/iobuffer.cc:402
|
#6 0x00007f07dbb59393 in read_skipping_prefixes (dst=0x7f0806f34e24, len=190, pos=<synthetic pointer>, file=0x7f07d1a9d100)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_file_read.cc:101
|
#7 pread_bin_internal (file=0x7f07d1a9d100, pos=352257, ret_ptr=ret_ptr@entry=0x7f07a97f8838, max_header_size=max_header_size@entry=0)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_file_read.cc:150
|
#8 0x00007f07dbb59774 in pread_compressed (file=<optimized out>, pos=<optimized out>, ret_ptr=ret_ptr@entry=0x7f07a97f88d8)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_file_read.cc:181
|
#9 0x00007f07dbb4cb47 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071e062f2c, start=start@entry=1474, end=end@entry=1476, dst=dst@entry=0x7f071e060ab4)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:445
|
#10 0x00007f07dbb4ccc5 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071e060a84, start=1474, start@entry=1404, end=end@entry=1506, dst=dst@entry=0x7f071e2cc1b0)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:610
|
#11 0x00007f07dbb4ccc5 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071e2cc180, start=1404, start@entry=0, end=end@entry=2186, dst=dst@entry=0x7f071e2cc0c8)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:610
|
#12 0x00007f07dbb4ccc5 in modify_node (rq=rq@entry=0x7f07a97f8dc0, nptr=nptr@entry=0x7f071d9480c0, start=start@entry=0, end=3146, dst=dst@entry=0x7f071e2cc010)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:610
|
#13 0x00007f07dbb4db90 in modify_btree (rq=rq@entry=0x7f07a97f8dc0, root=0x7f071d9480c0, errcode=errcode@entry=0x7f07a97f8c74)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/btree_modify.cc:768
|
#14 0x00007f07dbb5b6cc in update_indexes (save_callback_ctx=0x7f07a97f93e0, save_callback=0x7f07dbeec600 <saveDocsCallback(DocInfo const*, DocInfo const*, void*)>, numdocs=1573, idvals=<optimized out>,
|
ids=0x7f071bdb8260, seqvals=<optimized out>, seqs=<optimized out>, db=0x7f07d1a9d100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:238
|
#15 couchstore_save_documents_and_callback (db=<optimized out>, docs=<optimized out>, infos=<optimized out>, numdocs=<optimized out>, options=options@entry=3,
|
save_cb=save_cb@entry=0x7f07dbeec600 <saveDocsCallback(DocInfo const*, DocInfo const*, void*)>, save_cb_ctx=<optimized out>)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/couchstore/src/couch_save.cc:407
|
#16 0x00007f07dbef5140 in CouchKVStore::saveDocs (this=this@entry=0x7f07d1b91800, vbid=..., docs=..., docinfos=..., kvctx=..., collectionsFlush=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:2039
|
#17 0x00007f07dbef5d6d in CouchKVStore::commit2couchstore (this=0x7f07d1b91800, collectionsFlush=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:1942
|
#18 0x00007f07dbef6685 in CouchKVStore::commit (this=0x7f07d1b91800, collectionsFlush=...)
|
at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/couch-kvstore/couch-kvstore.cc:1209
|
#19 0x00007f07dbe21a29 in EPBucket::commit (this=0x7f07d1b5c300, kvstore=..., collectionsFlush=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:500
|
#20 0x00007f07dbe2276d in EPBucket::flushVBucket (this=0x7f07d1b5c300, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:434
|
#21 0x00007f07dbe66ab5 in Flusher::flushVB (this=0x7f07d1a6dcc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:300
|
#22 0x00007f07dbe66fdd in Flusher::step (this=0x7f07d1a6dcc0, task=0x7f07d1a733b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/flusher.cc:207
|
#23 0x00007f07dbe5deb8 in ExecutorThread::run (this=0x7f07d1a6d400) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:148
|
#24 0x00007f080b0fc0e7 in run (this=0x7f07d7dbe3d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:59
|
#25 platform_thread_wrap (arg=0x7f07d7dbe3d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:72
|
#26 0x00007f08092ecdc5 in start_thread () from /usr/lib64/libpthread-2.17.so
|
#27 0x00007f080901a21d in clone () from /usr/lib64/libc-2.17.so
|
(gdb)
|
cbcollect_info attached.
This is the first time we are running the test in madhatter.