Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45852

[Windows] - Minidumps seen during multi node rebalance out + collections drop/recreate

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • No

    Description

      Script to Repro

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_drop_recreate_rebalance.CollectionsDropRecreateRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_1000_collections,GROUP=set1'
      

      Steps to Repro
      1. Create a 5 node cluster
      ----------------------------------------------------------------------

      Nodes Services Version CPU Status

      ----------------------------------------------------------------------

      172.23.104.247 kv 7.0.0-4985-enterprise 3.48850775872 Cluster node
      172.23.105.1 None     <--- IN —
      172.23.105.137 None     <--- IN —
      172.23.105.183 None     <--- IN —
      172.23.106.116 None     <--- IN —

      ----------------------------------------------------------------------

      2. Create buckets/scopes/collections/data
      2021-04-21 13:44:12,509 | test | INFO | MainThread | [table_view:display:72] Bucket statistics
      ------------------------------------------------------------------------

      Bucket Type Replicas Durability TTL Items RAM Quota RAM Used Disk Used

      ------------------------------------------------------------------------

      bucket1 ephemeral 2 none 0 5000 5242880000 657141808 170
      default couchbase 2 none 0 5000 5242880000 502429160 356963799

      ------------------------------------------------------------------------

      3. Rebalance out 2 nodes (172.23.104.247 and 172.23.105.1)
      2021-04-21 13:44:12,700 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview
      ----------------------------------------------------------------------

      Nodes Services Version CPU Status

      ----------------------------------------------------------------------

      172.23.106.116 kv 7.0.0-4985-enterprise 6.35177252954 Cluster node
      172.23.104.247 kv 7.0.0-4985-enterprise 4.37340622344 — OUT --->
      172.23.105.137 kv 7.0.0-4985-enterprise 4.42507375123 Cluster node
      172.23.105.1 kv 7.0.0-4985-enterprise 4.24507075118 — OUT --->
      172.23.105.183 kv 7.0.0-4985-enterprise 7.23987933534 Cluster node

      ----------------------------------------------------------------------

      2021-04-21 13:45:58,575 | test | ERROR | pool-2-thread-27 | [rest_client:_rebalance_status_and_progress:1510]

      {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'5359c165f7e62f30159335bbc728d392', u'statusIsStale': False, u'status': u'notRunning'}

      - rebalance failed

      At this exact time we see a67cffcb-d052-4beb-b425-806ef6b57c98.dmp on 172.23.104.247.

      cbcollect_info attached.

      Attachments

        Issue Links

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty