Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49806

memcached restart during rebalance can result in some replications being missing for a long time (was Rebalance exited with reason bad_replicas)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • Morpheus
    • 6.6.4, 7.0.3, 7.1.0
    • ns_server
    • 7.1.0-1767-enterprise

    Description

      1. Step 1: Create a 4 node cluster
      2. Step 2: Create required buckets and collections.
      3. Step 3: Create 625000 items sequentially
      4. Step 4: Update 625000 RandonKey keys to create 50 percent fragmentation
      5. Step 5: Create 625000 items sequentially
      6. Step 6: Update 625000 RandonKey keys to create 50 percent fragmentation
      7. Step 7: Rebalance in with Loading of docs. Abort and resume rebl at 20%, 40%, 60%, 80%
      8. Step 8: Crash Magma/memc with Loading of docs
      9. Step 9: Rebalance Out with Loading of docs. Abort and resume rebl at 20%, 40%, 60%, 80%
      10. Step 10: Crash Magma/memc with Loading of docs
      11. Step 11: Rebalance In_Out with Loading of docs. Abort and resume rebl at 20%, 40%, 60%, 80%
      12. Step 12: Crash Magma/memc with Loading of docs
      13. Step 13: Swap with Loading of docs. Abort and resume rebl at 20%, 40%, 60%, 80%
      14. Step 14: Rebalance IN=2, OUT=1 with Loading of docs. Abort and resume rebl at 20%, 40%, 60%, 80%. Rebalance failed with bad_replicas. 

        {u'code': 2, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.110.69', u'tstamp': 1637980474681L, u'shortText': u'message', u'serverTime': u'2021-11-26T18:34:34.681Z', u'text': u"Bad replicators after rebalance:\nMissing = [{'ns_1@172.23.110.127','ns_1@172.23.110.69',256},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',257},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',258},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',259},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',260},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',261},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',262},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',263},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',264},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',265},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',266},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',267},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',268},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',269},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',270},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',271},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',272},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',273},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',274},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',275},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',276},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',277},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',278},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',279},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',280},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',281},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',282},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',283},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',284},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',285},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',286},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',287},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',288},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',289},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',290},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',291},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',292},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',293},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',294},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',295},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',296},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',297},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',298},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',299},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',300},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',301},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',302},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',303},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',304},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',305},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',306},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',307},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',308},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',309},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',310},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',311},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',312},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',313},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',314},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',315},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',316},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',317},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',318},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',319},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',320},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',321},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',322},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',323},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',324},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',325},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',326},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',327},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',328},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',329},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',330},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',331},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',332},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',333},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',334},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',335},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',336},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',337},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',338},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',339},\n           {'ns_1@172.23.110.127','ns_1@172.23.110.69',340},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',170},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',427},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',428},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',429},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',430},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',431},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',432},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',433},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',434},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',435},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',436},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',437},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',438},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',439},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',440},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',441},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',442},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',443},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',444},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',445},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',446},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',447},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',448},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',449},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',450},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',451},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',452},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',453},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',454},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',455},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',456},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',457},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',458},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',459},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',460},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',461},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',462},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',463},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',464},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',465},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',466},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',467},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',468},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',469},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',470},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',471},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',472},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',473},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',474},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',475},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',476},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',477},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',478},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',479},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',480},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',481},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',482},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',483},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',484},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',485},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',486},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',487},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',488},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',489},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',490},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',491},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',492},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',493},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',494},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',495},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',496},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',497},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',498},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',499},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',500},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',501},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',502},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',503},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',504},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',505},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',506},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',507},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',508},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',509},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',510},\n           {'ns_1@172.23.110.128','ns_1@172.23.110.69',511},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',0},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',1},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',2},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',3},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',4},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',5},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',6},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',7},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',8},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',9},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',10},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',11},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',12},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',13},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',14},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',15},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',16},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',17},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',18},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',19},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',20},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',21},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',22},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',23},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',24},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',25},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',26},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',27},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',28},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',29},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',30},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',31},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',32},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',33},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',34},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',35},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',36},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',37},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',38},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',39},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',40},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',41},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',42},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',43},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',44},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',45},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',46},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',47},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',48},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',49},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',50},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',51},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',52},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',53},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',54},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',55},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',56},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',57},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',58},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',59},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',60},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',61},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',62},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',63},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',64},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',65},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',66},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',67},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',68},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',69},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',70},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',71},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',72},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',73},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',74},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',75},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',76},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',77},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',78},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',79},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',80},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',81},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',82},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',83},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.127',84},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',341},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',342},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',343},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',344},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',345},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',346},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',347},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',348},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',349},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',350},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',351},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',352},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',353},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',354},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',355},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',356},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',357},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',358},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',359},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',360},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',361},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',362},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',363},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',364},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',365},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',366},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',367},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',368},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',369},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',370},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',371},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',372},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',373},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',374},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',375},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',376},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',377},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',378},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',379},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',380},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',381},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',382},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',383},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',384},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',385},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',386},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',387},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',388},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',389},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',390},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',391},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',392},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',393},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',394},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',395},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',396},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',397},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',398},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',399},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',400},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',401},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',402},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',403},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',404},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',405},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',406},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',407},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',408},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',409},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',410},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',411},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',412},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',413},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',414},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',415},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',416},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',417},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',418},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',419},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',420},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',421},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',422},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',423},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',424},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',425},\n           {'ns_1@172.23.110.69','ns_1@172.23.110.128',426}]\nExtras = []"}


        guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job2.ini -p bucket_storage=magma,bucket_eviction_policy=fullEviction,rerun=False -t aGoodDoctor.Hospital.Murphy.SystemTestMagma,nodes_init=2,graceful=True,skip_cleanup=True,num_items=25000000,num_buckets=1,bucket_names=GleamBook,doc_size=1024,bucket_type=membase,eviction_policy=fullEviction,iterations=10,batch_size=1000,sdk_timeout=60,log_level=debug,infra_log_level=debug,rerun=False,skip_cleanup=True,key_size=18,randomize_doc_size=False,randomize_value=True,assert_crashes_on_load=True,num_collections=50,maxttl=10,num_indexes=50,pc=25,index_nodes=0,cbas_nodes=0,fts_nodes=0,ops_rate=60000,ramQuota=2048,doc_ops=create:update:delete:read,rebl_ops_rate=40000,key_type=RandomKey,vbuckets=512,mutation_perc=100 -m rest'

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-49217 vbuckets can go unreplicated for a long time if a replication connection between nodes is down during rebalance

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              neelima.premsankar Neelima Premsankar
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty