Details
-
Bug
-
Resolution: Fixed
-
Critical
-
4.6.0, 5.0.0
-
Untriaged
-
Centos 64-bit
-
-
Unknown
Description
Setup:
- 5 source nodes, 4 target nodes
- 1 bucket
- 1B documents
- sourceNozzlePerNode = 4, targetNozzlePerNode = 64
Source of panic:
if docKey == "" {
|
panic(fmt.Sprintf("%v unique-key= %v, Empty docKey, req=%v, bigDoc_map=%v", xmem.Id(), key, originalReq.Req, bigDoc_map))
|
}
|
Actual crash:
panic: xmem_15163b825c80888bab3bedce5a0709ff/bucket-1/bucket-1_172.23.96.102:11210_0 unique-key= symmetric-000002187032, Empty docKey, req={MCRequest opcode=GET, bodylen=0, key=''}, bigDoc_map=map[symmetric-000002887701:0xc8237f6370 symmetric-000002983014:0xc820fe1f40 symmetric-000003066794:0xc8243daa50 symmetric-000002881431:0xc8234814a0 symmetric-000002882610:0xc823fb2d70 symmetric-000002889654:0xc823480d20 symmetric-000002968862:0xc821e061e0 symmetric-000002893789:0xc821b99040 symmetric-000003009547:0xc821f1dc20 symmetric-000003031660:0xc822292460 symmetric-000002863726:0xc825283590 symmetric-000002887073:0xc825282b40 symmetric-000002896698:0xc822dd42d0 symmetric-000002996998:0xc822f2c4b0 symmetric-000002879189:0xc821235220 symmetric-000002887691:0xc82619ee10 symmetric-000002893619:0xc8229acf00 symmetric-000002943069:0xc82619e960 symmetric-000002988722:0xc822293270 symmetric-000003086944:0xc8224135e0 symmetric-000002858620:0xc8237f7720 symmetric-000002884252:0xc821fce9b0 symmetric-000002868180:0xc82149a500 symmetric-000002964560:0xc82412a910 symmetric-000002989954:0xc82615e730 symmetric-000003050294:0xc823fb2050 symmetric-000002900085:0xc822c57130 symmetric-000002975569:0xc8235b14a0 symmetric-000002906557:0xc821adb810 symmetric-000002951241:0xc823fb3590 symmetric-000003084483:0xc824a61d60 symmetric-000002900115:0xc82096b6d0 symmetric-000003057930:0xc82499ceb0 symmetric-000002973659:0xc8258f0d20 symmetric-000003004633:0xc82324e000 symmetric-000003039595:0xc82499c730 symmetric-000003068751:0xc824c28e10 symmetric-000003093158:0xc823ad1310 symmetric-000002888822:0xc821234f50 symmetric-000002967033:0xc8213d0be0 symmetric-000003081592:0xc8213d1cc0 symmetric-000002865364:0xc8205ab950 symmetric-000003069927:0xc822ea4d20 symmetric-000002187032:0xc823480960 symmetric-000002936585:0xc822412eb0 symmetric-000002954422:0xc821fcf680 symmetric-000002985324:0xc8224123c0 symmetric-000003037550:0xc820b881e0 symmetric-000002865416:0xc82403efa0 symmetric-000002931953:0xc824847310 symmetric-000002958920:0xc82148f220 symmetric-000002964382:0xc82328b900 symmetric-000002966937:0xc8213d00f0 symmetric-000003002493:0xc821f28d20 symmetric-000002856775:0xc824190320 symmetric-000002905776:0xc820ae4370 symmetric-000002855226:0xc823622460 symmetric-000002903334:0xc822c639a0 symmetric-000002952712:0xc82403f680 symmetric-000002962122:0xc8240405a0 symmetric-000003049168:0xc82148e280 symmetric-000002876848:0xc821699e50 symmetric-000002910818:0xc821adacd0 symmetric-000003002503:0xc823480000 symmetric-000003063715:0xc823cb94a0 symmetric-000003016369:0xc82154da90 symmetric-000002857893:0xc824851770 symmetric-000002901811:0xc824715f40 symmetric-000002980547:0xc8237a4280 symmetric-000003007582:0xc822412c80 symmetric-000002853086:0xc823723680 symmetric-000003053657:0xc8235b0500 symmetric-000003060534:0xc823481a40 symmetric-000002988050:0xc82149abe0 symmetric-000003046939:0xc823fb2780 symmetric-000002886977:0xc823b88a50 symmetric-000002969614:0xc820139400 symmetric-000002980235:0xc821b98b90 symmetric-000003001722:0xc824847400 symmetric-000003031112:0xc8219465f0 symmetric-000002860275:0xc824b574f0 symmetric-000002881343:0xc821f72dc0 symmetric-000003013278:0xc82557a500 symmetric-000002895529:0xc8235b6410 symmetric-000002985456:0xc8215d45a0 symmetric-000002884520:0xc823b89770 symmetric-000002987991:0xc8248e1770 symmetric-000002993889:0xc822f2de00 symmetric-000003010059:0xc825767360 symmetric-000003082623:0xc821f72640 symmetric-000002866637:0xc823fb2d20 symmetric-000002867841:0xc8212a7680 symmetric-000003055215:0xc8205aa140 symmetric-000002993919:0xc82149a9b0 symmetric-000003095268:0xc825161b80 symmetric-000002855554:0xc824abd220 symmetric-000002983184:0xc822c62e10 symmetric-000003067872:0xc822459860 symmetric-000003082151:0xc824b4fea0 symmetric-000002951533:0xc824a607d0 symmetric-000002967741:0xc822c62370 symmetric-000002983766:0xc824ff8d70 symmetric-000003062963:0xc8222d6d20 symmetric-000003066604:0xc8212a7770 symmetric-000003081402:0xc824e74ff0 symmetric-000002901981:0xc821e0be00 symmetric-000002949929:0xc8240400a0 symmetric-000002969166:0xc8235b0e10 symmetric-000003035897:0xc823a6db30 symmetric-000003055385:0xc82499dea0 symmetric-000002952060:0xc8229b0d20 symmetric-000002961471:0xc823a6c2d0 symmetric-000002905004:0xc8243f83c0 symmetric-000003056034:0xc826385ef0 symmetric-000002957603:0xc821499f90 symmetric-000003039405:0xc824d39590 symmetric-000002869884:0xc82224e780 symmetric-000002904890:0xc820ae4eb0 symmetric-000002962650:0xc823a6cd20 symmetric-000002987801:0xc821a545a0 symmetric-000003004141:0xc824fea780 symmetric-000003043828:0xc823480730 symmetric-000003063067:0xc8229b13b0 symmetric-000002859856:0xc823857b80 symmetric-000002963826:0xc824851590 symmetric-000002957171:0xc820086af0 symmetric-000002858152:0xc825767720 symmetric-000002865586:0xc821e06f00 symmetric-000003000954:0xc825b4b810 symmetric-000002882780:0xc8225759a0 symmetric-000002889126:0xc8224582d0 symmetric-000002986095:0xc824100780 symmetric-000003050304:0xc825690a50 symmetric-000003089115:0xc823878690 symmetric-000002862950:0xc825b4a000 symmetric-000002929549:0xc822340140 symmetric-000002982880:0xc82224c780 symmetric-000003026559:0xc825c80690 symmetric-000003035907:0xc823ad05f0 symmetric-000003055567:0xc821e06870 symmetric-000002908402:0xc824847220 symmetric-000002915899:0xc8243ed630 symmetric-000003034193:0xc8212a62d0 symmetric-000003089085:0xc823623720 symmetric-000003031082:0xc823ad15e0 symmetric-000003034771:0xc822c623c0 symmetric-000003025778:0xc8263854a0 symmetric-000003032441:0xc8219479a0 symmetric-000002933276:0xc824ae1a40 symmetric-000002935146:0xc82148fa90 symmetric-000002964212:0xc82557a3c0 symmetric-000002970478:0xc8222939f0 symmetric-000002910988:0xc821ada190 symmetric-000002863054:0xc8257674f0 symmetric-000002904900:0xc820ae4910 symmetric-000002936367:0xc8224139f0 symmetric-000003015148:0xc8212a6ff0 symmetric-000003058161:0xc824abd450 symmetric-000002852982:0xc82148e000 symmetric-000002879019:0xc8225742d0 symmetric-000002956875:0xc82403e550 symmetric-000002986105:0xc825766910 symmetric-000003056746:0xc824191040 symmetric-000002856197:0xc824040c80 symmetric-000002869914:0xc823a7d4a0 symmetric-000003023448:0xc82499d540 symmetric-000003060246:0xc82017d900 symmetric-000003058783:0xc822c62780 symmetric-000003059865:0xc82310ab40 symmetric-000002860497:0xc82224ef50 symmetric-000002906225:0xc8212342d0 symmetric-000002969784:0xc8248a35e0 symmetric-000003002271:0xc821499630 symmetric-000003007412:0xc8213d11d0 symmetric-000003030816:0xc8213d1770 symmetric-000003084261:0xc821adaaf0 symmetric-000002908370:0xc821adb270 symmetric-000002933494:0xc82148e3c0 symmetric-000002930725:0xc8243ec550 symmetric-000002996808:0xc820b890e0 symmetric-000003037222:0xc8243ed810 symmetric-000003087732
|
XDCR was aborted multiple times on two nodes:
- 172.23.96.106
- 172.23.96.107
Replication eventually succeeded after several crashes and restarts.