Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
7.0.0-2017
-
Untriaged
-
-
1
-
Unknown
Description
Volume test steps:
Step 1: Create a 3 node cluster
Step 2: Create 1 magma bucket 1 scope and 1 collection.
Step 3: Create 10M items and Delete everything. checkout fragmentation
Step 4: Load 20M items, sequential keys
Step 5: Update the first set of 1M items 10 times
Step 6: Reverse Update last set of 1M items 10 times
Step 7: Create random keys
Step 8: Update all random 10 times
Step 9: Delete/Re-Create all random items
Rate of deleting keys is:
+------------+--------------------------------------------+------------+
|
| Min | Trend | Max |
|
+------------+--------------------------------------------+------------+
|
| 0.000 | **....*...............................X | 196612.388 |
|
| 193017.982 | ................................*.......X | 205583.417 |
|
| 197122.877 | ..................................*.X | 197739.261 |
|
| 196306.693 | ..................................*....X | 205034.965 |
|
| 201389.610 | ....................................X | 201389.610 |
|
| 191704.296 | ................................*...*.X | 199575.425 |
|
| 192327.672 | ................................*....X | 199929.071 |
|
| 193277.722 | .................................*...X | 199650.350 |
|
| 199466.533 | ....................................X | 199466.533 |
|
| 194021.978 | .................................*....X | 202434.565 |
|
| 189827.173 | ...............................*..*..X | 198765.235 |
|
| 197000.999 | ..................................*...X | 202482.517 |
|
| 199061.938 | ...................................*.X | 199721.279 |
|
| 196709.291 | ..................................*...X | 202583.417 |
|
| 200347.652 | ....................................X | 200347.652 |
|
| 193472.527 | .................................*.....X | 205382.617 |
|
| 204983.017 | ......................................X | 204983.017 |
|
| 194671.329 | .................................*..X | 198937.063 |
|
| 198559.441 | ...................................X | 198559.441 |
|
| 195910.090 | ..................................X | 195910.090 |
|
| 191003.996 | ...............................*..X | 194003.996 |
|
| 191221.778 | ................................X | 191221.778 |
|
| 190142.857 | ...............................*X | 190565.435 |
|
+------------+--------------------------------------------+------------+
|
Step 10: Start validating the deleted keys. Delete information may not have persisted.
While reading core dumps are found and able to read data also.
Few deleted keys that test is able to read:
NgggXZjgc-30456372,07PqhQGBH-30456472,5iHmnzggX-30456529,UpIeMEtn2-30456660,72z1VPCHm-30456186,hXODDzsGF-30456081,CD0A7Ksej-30456836,jSJpcTcsf-30456704,qbItE0Wat-30456254,qNRLv2ii6-30456889,B0L3mxuMe-30456999,UpIeMEtn2-30456665,Qk6Ij4dea-30456050,K3xuOXDSO-30456442,CD0A7Ksej-30456839,VPCHmixsU-30456170,JUQxIwVCM-30456823,Kck0Ql1P8-30456236,lrxKrxann-30456224,Kck0Ql1P8-30456237,UKnfsd3Nf-30456064,UKnfsd3Nf-30456061,qhQGBHQN2-30456470,kYO2G7qQF-30456592,Zu3NQAdVO-30456365,0L3mxuMe6-30456995,O6LNmj2Qd-30456943,pTSMYoiMv-30456281,Zu3NQAdVO-30456364,NHFzfGUNe-30456974,nxNHl2qph-30456406,BKmG1cXqa-30456346,nxNHl2qph-30456403,A5mgwvUPV-30456778,QpXATVPuY-30456359,P7jSl0fZS-30456933,y7D4WBef6-30456517,y7D4WBef6-30456519,GU5axhg64-30456408,krwzoa4cd-30456488,46qydnEeC-30456719,8sNcTmuKC-30456113,8sNcTmuKC-30456116,ydnEeCORB-30456723,1Zu3NQAdV-30456037,NRLv2ii6a-30456883,NRLv2ii6a-30456886,gcpYEvnnC-30456507,NcTmuKCPO-30456112,xYH7IrQyP-30456101,nzggXS1TH-30456533,cMGGGeOlQ-30456828,cMGGGeOlQ-30456826,k0Ql1P8rW-30456232,FTOBqd2ma-30456419,FTOBqd2ma-30456418,k0Ql1P8rW-30456233,0swqueufk-30456076,0swqueufk-30456078,SwRydsoAK-30456879,fpqRe4qPb-30456702,CJAwDFDCk-30456437,V4rEM6Kx2-30456304,uRgGq5WHX-30456607,xNbGgCCR9-30456549,wzoa4cdpQ-30456484,ydnEeCORB-30456717,7D4WBef6s-30456527,nEeCORBA7-30456721,QbAld3ADW-30456908,QbAld3ADW-30456903,MWp1AN5Jk-30456493,8EII1IzhP-30456268,8EII1IzhP-30456267,PkYO2G7qQ-30456593,PkYO2G7qQ-30456596,mg8l3xRHv-30456318,mg8l3xRHv-30456317,CJbsOR46q-30456731,5NZ2KlFoi-30456071,MHVI43gYK-30456387,SqMntUBJK-30456919,S3Tt0y3gG-30456861,SqMntUBJK-30456917,HynNVYxeg-30456692,MI93QPzri-30456153,II1IzhPGx-30456263,OBqd2maxl-30456428,II1IzhPGx-30456266,0A7Ksej1W-30456846,KL9U11xpg-30456741,pD6yysuqr-30456953,f2hOEdnk2-30456643,g8sNcTmuK-30456119,Ke0f7oHrg-30456242,AwDFDCk1J-30456432,AwDFDCk1J-30456433,lxmfSA7ZF-30456582,2hOEdnk2R-30456640,On7tKpFxI-30456029,SUpIeMEtn-30456666,TTcwVUFTQ-30456562,KmG1cXqaY-30456343,EeCORBA7Z-30456720,P0NgggXZj-30456374,DqsYhZwOd-30456049,vNHFzfGUN-30456978,7g8sNcTmu-30456117,buil2CsHO-30456137,sUHminpcx-30456179,buil2CsHO-30456132,1VPCHmixs-30456184,SzoKo6akp-30456121,ewoJt2jsr-30456603,I1IzhPGxS-30456260,wvUPVCi7v-30456772,huQbuil2C-30456139,GGGeOlQlM-30456834,mnzggXS1T-30456538,8YyDPR3sl-30456635,wbO6LNmj2-30456936,HFzfGUNeJ-30456973,IqbItE0Wa-30456258,tyQk6Ij4d-30456054,jZKe0f7oH-30456245,IqbItE0Wa-30456253,93QPzrijC-30456152,eBE6gC7bY-30456457,sqHynNVYx-30456697,eBE6gC7bY-30456456,sqHynNVYx-30456696,YhZwOdY5E-30456042,uOXDSOTe6-30456390,uuUKnfsd3-30456065,u3NQAdVOA-30456360,2zC5NZ2Kl-30456000,Og8YyDPR3-30456959,rgqmoXv3K-30456857,rgqmoXv3K-30456856,VI43gYKBp-30456383,RXRg0JgSs-30456084,Y07PqhQGB-30456479,Y07PqhQGB-30456477,cJUQxIwVC-30456822,zC5NZ2KlF-30456001,rwzoa4cdp-30456489,yysuqrXwb-30456951,SJpcTcsfz-30456712,suF9bO7W1-30456574,suF9bO7W1-30456571,3rPeBE6gC-30456459,tuuUKnfsd-30456069,SztL4ZaUC-30456010,bGgCCR95i-30456544,wSUpIeMEt-30456667,ntUBJKbkx-30456920,wSUpIeMEt-30456669,F9bO7W1yY-30456570,CxYH7IrQy-30456104,bKo4nXLOC-30456210,bO6LNmj2Q-30456935,2KgWEGPg2-30456786,oE75Fv5bL-30456810,gXcyckoen-30456143,r46ZTlpw3-30456462,KFbKo4nXL-30456215,pIeMEtn2y-30456662,Vb49bLyDp-30456651,dqwqeDSci-30456684,dqwqeDSci-30456681,ztL4ZaUCj-30456012,gXcyckoen-30456148,ATVPuYtEt-30456353,rv3B1tH8k-30456555,gGq5WHXLu-30456611,rv3B1tH8k-30456550,JSztL4ZaU-30456014,aEwLaRapk-30456324,qSzoKo6ak-30456127,wDFDCk1JF-30456435,8r4QV3LkD-30456501,mc9ezuOVM-30456894,I6LGWVd8y-30456161,I6LGWVd8y-30456160,Bqd2maxlG-30456421,BcdqwqeDS-30456686,yDPR3sllT-30456632,sYhZwOdY5-30456045,sYhZwOdY5-30456040,UlxmfSA7Z-30456589,Tt0y3gG0J-30456860,pcTcsfzhk-30456710,tFPxI2QQ8-30456795,D0A7Ksej1-30456838,eF87GLb99-30456618,I93QPzrij-30456155,EwLaRapkR-30456321,I93QPzrij-30456150,VPkYO2G7q-30456594,oe8r4QV3L-30456505,pP0NgggXZ-30456377,DFDCk1JFT-30456431,cGWKFbKo4-30456209,N2r46ZTlp-30456466,yY2TTcwVU-30456568,vrv3B1tH8-30456554,EvnnCeHFt-30456512,VpMWp1AN5-30456495,vjK3xuOXD-30456446,5thXODDzs-30456097,vjK3xuOXD-30456443,pYEvnnCeH-30456513,Df2hOEdnk-30456647,rPeBE6gC7-30456458,2r46ZTlpw-30456465,HmnzggXS1-30456535,xhg64UY2S-30456413,gKL9U11xp-30456745,gKL9U11xp-30456744,ydsoAK1TB-30456870,Df2hOEdnk-30456642,ydsoAK1TB-30456871,CRw5qq8N8-30456677,RgGq5WHXL-30456614,U5thXODDz-30456095,yQk6Ij4de-30456051,hvNHFzfGU-30456976,1Zu3NQAdV-30456362,1Zu3NQAdV-30456363,icmg8l3xR-30456308,icmg8l3xR-30456309,PLapTSMYo-30456289,92KgWEGPg-30456785,Qbuil2CsH-30456135,wqueufkDt-30456075,KcJUQxIwV-30456814,OHCxYH7Ir-30456107,OHCxYH7Ir-30456106,HCRw5qq8N-30456675,T2MHVI43g-30456389,yIzwVb49b-30456657,Y2TTcwVUF-30456564,YH7IrQyPn-30456100,oU5thXODD-30456099,YyDPR3sll-30456631,EbF1g9pr9-30456801,TOBqd2max-30456425,Og8YyDPR3-30456628,MyKcJUQxI-30456818,A7Ksej1Wr-30456842,cdqwqeDSc-30456687,D4WBef6sp-30456521,GJBlpOvuC-30456337,GJBlpOvuC-30456338,62zC5NZ2K-30456005,8l3xRHv2m-30456313
|
Stack Trace: 172.23.97.39: Stack Trace of c525dd64-c774-45f5-5e24fcb3-b8d9d831.dmp
rm -rf /opt/couchbase/var/lib/couchbase/crash/c525dd64-c774-45f5-5e24fcb3-b8d9d831.core
|
//opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/c525dd64-c774-45f5-5e24fcb3-b8d9d831.dmp > /opt/couchbase/var/lib/couchbase/crash/c525dd64-c774-45f5-5e24fcb3-b8d9d831.core
|
gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/c525dd64-c774-45f5-5e24fcb3-b8d9d831.core -ex bt all -ex quit
|
Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'.
|
#0 0x00007fbeb30d383d in std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#0 0x00007fbeb30d383d in std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::_Rb_tree_node_base&) () from /opt/couchbase/bin/../lib/libstdc++.so.6
|
#1 0x00007fbeb6897ee2 in std::pair<std::_Rb_tree_iterator<std::pair<unsigned long const, std::pair<std::shared_ptr<GlobalTask>, TaskQueue*> > >, bool> std::_Rb_tree<unsigned long, std::pair<unsigned long const, std::pair<std::shared_ptr<GlobalTask>, TaskQueue*> >, std::_Select1st<std::pair<unsigned long const, std::pair<std::shared_ptr<GlobalTask>, TaskQueue*> > >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, std::pair<std::shared_ptr<GlobalTask>, TaskQueue*> > > >::_M_insert_unique<std::pair<unsigned long, std::pair<std::shared_ptr<GlobalTask>, TaskQueue*> > >(std::pair<unsigned long, std::pair<std::shared_ptr<GlobalTask>, TaskQueue*> >&&) () from /opt/couchbase/bin/../lib/libep.so
|
#2 0x00007fbeb689486d in ExecutorPool::_schedule(std::shared_ptr<GlobalTask>) () from /opt/couchbase/bin/../lib/libep.so
|
#3 0x00007fbeb689499d in ExecutorPool::schedule(std::shared_ptr<GlobalTask>) () from /opt/couchbase/bin/../lib/libep.so
|
#4 0x00007fbeb68bd90a in KVBucket::visitAsync(std::unique_ptr<PausableVBucketVisitor, std::default_delete<PausableVBucketVisitor> >, char const*, TaskId, std::chrono::duration<long, std::ratio<1l, 1000000l> >) () from /opt/couchbase/bin/../lib/libep.so
|
#5 0x00007fbeb6843701 in DurabilityTimeoutTask::run() () from /opt/couchbase/bin/../lib/libep.so
|
#6 0x00007fbeb68a3083 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so
|
#7 0x00007fbeb689b92f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so
|
#8 0x00007fbeb52c2827 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0
|
#9 0x00007fbeb28ffe25 in start_thread () from /lib64/libpthread.so.0
|
#10 0x00007fbeb2629bad in clone () from /lib64/libc.so.6
|
Memcahced Logs:
172.23.97.39 : Found message in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000050.txt
|
2020-05-14T10:40:21.403185-07:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-2017). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/c525dd64-c774-45f5-5e24fcb3-b8d9d831.dmp before terminating.
|
2020-05-14T10:40:21.403238-07:00 CRITICAL Stack backtrace of crashed thread:
|
2020-05-14T10:40:21.416765-07:00 CRITICAL /opt/couchbase/bin/memcached() [0x400000+0x13946d]
|
2020-05-14T10:40:21.416792-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ea) [0x400000+0x14f1ba]
|
2020-05-14T10:40:21.416806-07:00 CRITICAL /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xb8) [0x400000+0x14f4f8]
|
2020-05-14T10:40:21.416820-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fbeb28f8000+0xf6d0]
|
2020-05-14T10:40:21.416875-07:00 CRITICAL /opt/couchbase/bin/../lib/libstdc++.so.6(_ZSt29_Rb_tree_insert_and_rebalancebPSt18_Rb_tree_node_baseS0_RS_+0xcd) [0x7fbeb302d000+0xa683d]
|
2020-05-14T10:40:21.416897-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fbeb675d000+0x13aee2]
|
2020-05-14T10:40:21.416911-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fbeb675d000+0x13786d]
|
2020-05-14T10:40:21.416920-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fbeb675d000+0x13799d]
|
2020-05-14T10:40:21.416930-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fbeb675d000+0x16090a]
|
2020-05-14T10:40:21.416944-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fbeb675d000+0xe6701]
|
2020-05-14T10:40:21.416967-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fbeb675d000+0x146083]
|
2020-05-14T10:40:21.416972-07:00 CRITICAL /opt/couchbase/bin/../lib/libep.so() [0x7fbeb675d000+0x13e92f]
|
2020-05-14T10:40:21.416976-07:00 CRITICAL /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7fbeb52b2000+0x10827]
|
2020-05-14T10:40:21.416982-07:00 CRITICAL /lib64/libpthread.so.0() [0x7fbeb28f8000+0x7e25]
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-