Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44194

Couchstore: Rebalance failed due to bad replicas.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 7.0.0
    • Cheshire-Cat
    • couchbase-bucket
    • 7.0.0-4413
    • Untriaged
    • 1
    • No

    Description

      1. Create a 15 node cluster
      2. Create required buckets and collections.
      3. Create 100000000 items sequentially
      4. Rebalance IN with Loading of docs
      5. Rebalance OUT with Loading of docs
      6. Rebalance SWAP with Loading of docs
      7. Rebalance IN/OUT with Loading of docs
      8. Rebalance OUT/IN with Loading of docs
      9. Rebalance IN with Loading of docs
      10. Rebalance OUT with Loading of docs
      11. Rebalance SWAP with Loading of docs
      12. Rebalance IN/OUT with Loading of docs. Rebalance failed.

      {u'code': 2, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.120.170', u'tstamp': 1612814859680L, u'shortText': u'message', u'serverTime': u'2021-02-08T12:07:39.680Z', u'text': u"Bad replicators after rebalance:\nMissing = [{'ns_1@172.23.120.170','ns_1@172.23.121.115',255},\n           {'ns_1@172.23.120.170','ns_1@172.23.121.115',256},\n           {'ns_1@172.23.120.170','ns_1@172.23.121.115',257},\n           {'ns_1@172.23.120.170','ns_1@172.23.121.115',258},\n           {'ns_1@172.23.120.170','ns_1@172.23.121.115',366},\n           {'ns_1@172.23.120.170','ns_1@172.23.121.115',436},\n           {'ns_1@172.23.120.170','ns_1@172.23.121.115',437},\n           {'ns_1@172.23.120.170','ns_1@172.23.121.115',438},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',214},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',215},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',216},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',217},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',218},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',519},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',670},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.124',987},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',219},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',220},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',262},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',577},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',702},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',703},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',925},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.126',931},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',152},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',153},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',425},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',585},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',730},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',789},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',790},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.129',791},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',265},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',350},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',884},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',910},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',911},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',912},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',996},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.130',997},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.131',154},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.131',155},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.131',156},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.131',157},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.131',158},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.131',264},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.131',586},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.133',168},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.133',493},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.133',494},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.133',495},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.133',496},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.133',497},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.133',840},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.135',169},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.135',170},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.135',171},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.135',486},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.135',728},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.135',804},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.135',805},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.136',207},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.136',208},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.136',209},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.136',210},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.136',211},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.136',998},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.136',999},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.139',28},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.139',29},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.139',544},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.139',545},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.139',711},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.139',712},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.139',718},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.140',212},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.140',241},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.140',413},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.140',424},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.140',553},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.140',757},\n           {'ns_1@172.23.121.115','ns_1@172.23.121.140',923},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',266},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',290},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',291},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',582},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',601},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',715},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',734},\n           {'ns_1@172.23.121.130','ns_1@172.23.121.115',926},\n           {'ns_1@172.23.121.131','ns_1@172.23.121.115',180},\n           {'ns_1@172.23.121.131','ns_1@172.23.121.115',181},\n           {'ns_1@172.23.121.131','ns_1@172.23.121.115',182},\n           {'ns_1@172.23.121.131','ns_1@172.23.121.115',452},\n           {'ns_1@172.23.121.131','ns_1@172.23.121.115',894},\n           {'ns_1@172.23.121.131','ns_1@172.23.121.115',902},\n           {'ns_1@172.23.121.131','ns_1@172.23.121.115',1015},\n           {'ns_1@172.23.121.133','ns_1@172.23.121.115',34},\n           {'ns_1@172.23.121.133','ns_1@172.23.121.115',268},\n           {'ns_1@172.23.121.133','ns_1@172.23.121.115',296},\n           {'ns_1@172.23.121.133','ns_1@172.23.121.115',465},\n           {'ns_1@172.23.121.133','ns_1@172.23.121.115',602},\n           {'ns_1@172.23.121.133','ns_1@172.23.121.115',690},\n           {'ns_1@172.23.121.133','ns_1@172.23.121.115',927},\n           {'ns_1@172.23.121.135','ns_1@172.23.121.115',229},\n           {'ns_1@172.23.121.135','ns_1@172.23.121.115',234},\n           {'ns_1@172.23.121.135','ns_1@172.23.121.115',267},\n           {'ns_1@172.23.121.135','ns_1@172.23.121.115',576},\n           {'ns_1@172.23.121.135','ns_1@172.23.121.115',955},\n           {'ns_1@172.23.121.135','ns_1@172.23.121.115',970},\n           {'ns_1@172.23.121.135','ns_1@172.23.121.115',983},\n           {'ns_1@172.23.121.136','ns_1@172.23.121.115',461},\n           {'ns_1@172.23.121.136','ns_1@172.23.121.115',773},\n           {'ns_1@172.23.121.136','ns_1@172.23.121.115',816},\n           {'ns_1@172.23.121.136','ns_1@172.23.121.115',852},\n           {'ns_1@172.23.121.136','ns_1@172.23.121.115',1008},\n           {'ns_1@172.23.121.139','ns_1@172.23.121.115',470},\n           {'ns_1@172.23.121.139','ns_1@172.23.121.115',538},\n           {'ns_1@172.23.121.139','ns_1@172.23.121.115',539},\n           {'ns_1@172.23.121.139','ns_1@172.23.121.115',540},\n           {'ns_1@172.23.121.139','ns_1@172.23.121.115',554},\n           {'ns_1@172.23.121.139','ns_1@172.23.121.115',555},\n           {'ns_1@172.23.121.139','ns_1@172.23.121.115',556},\n           {'ns_1@172.23.121.140','ns_1@172.23.121.115',302},\n           {'ns_1@172.23.121.140','ns_1@172.23.121.115',573},\n           {'ns_1@172.23.121.140','ns_1@172.23.121.115',726},\n           {'ns_1@172.23.121.140','ns_1@172.23.121.115',727},\n           {'ns_1@172.23.121.140','ns_1@172.23.121.115',731},\n           {'ns_1@172.23.121.140','ns_1@172.23.121.115',794},\n           {'ns_1@172.23.121.140','ns_1@172.23.121.115',1006},\n           {'ns_1@172.23.121.141','ns_1@172.23.121.115',42},\n           {'ns_1@172.23.121.141','ns_1@172.23.121.115',890},\n           {'ns_1@172.23.121.141','ns_1@172.23.121.115',891},\n           {'ns_1@172.23.121.141','ns_1@172.23.121.115',892},\n           {'ns_1@172.23.121.141','ns_1@172.23.121.115',893},\n           {'ns_1@172.23.121.141','ns_1@172.23.121.115',899},\n           {'ns_1@172.23.121.141','ns_1@172.23.121.115',900},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',14},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',15},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',16},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',17},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',18},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',47},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',144},\n           {'ns_1@172.23.121.48','ns_1@172.23.121.115',479}]\nExtras = []"}

      Attachments

        18. consoleText.txt
          962 kB
          Ritesh Agarwal

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-44097 Crash when collection disk size underflows with concurrent flush & compaction

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ritesh.agarwal Ritesh Agarwal
              ritesh.agarwal Ritesh Agarwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty