Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32782

[high-bucket] - rebalance is very slow after failover

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build 6.5.0-2082

      Observed that in high bucket density test(with 30 buckets), rebalance after hard fail over of kv node is very slow.
      In the test, it is 78% complete on kv nodes after ~14 hours. Please investigate if it is expected.

      Note: Removing kv node(from 4 node to 3 node) from same cluster without failover and rebalance takes ~206 min.

      Test:
      Out of 3 kv nodes, 1 is hard failed over and then rebalance started without adding node back.
      Buckets and docs: 32 buckets ~1M docs of 1KB per bucket.
      Number of replicas: 1
      XDCR: on
      KV ops: ~200 for entire cluster
      Cluster also had index, query, fts, eventing and analytics nodes.

      Logs-
      https://s3.amazonaws.com/bugdb/jira/mh_high_bkt_density_failover/collectinfo-2019-01-23T055648-ns_1%40172.23.97.12.zip
      https://s3.amazonaws.com/bugdb/jira/mh_high_bkt_density_failover/collectinfo-2019-01-23T055648-ns_1%40172.23.97.13.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          mahesh.mandhare Mahesh Mandhare (Inactive) created issue -
          ajit.yagaty Ajit Yagaty [X] (Inactive) made changes -
          Field Original Value New Value
          Assignee Ajit Yagaty [ ajit.yagaty ] Poonam Dhavale [ poonam ]
          raju Raju Suravarjjala made changes -
          Fix Version/s Mad-Hatter [ 15037 ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          mahesh.mandhare Mahesh Mandhare (Inactive) made changes -
          Assignee Mahesh Mandhare [ mahesh.mandhare ] Poonam Dhavale [ poonam ]
          poonam Poonam Dhavale made changes -
          Assignee Poonam Dhavale [ poonam ] Mahesh Mandhare [ mahesh.mandhare ]
          wayne Wayne Siu made changes -
          Summary rebalance is very slow after failover [high-bucket] - rebalance is very slow after failover
          Aliaksey Artamonau Aliaksey Artamonau (Inactive) made changes -
          Resolution Incomplete [ 4 ]
          Status Open [ 1 ] Resolved [ 5 ]
          raju Raju Suravarjjala made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            mahesh.mandhare Mahesh Mandhare (Inactive)
            mahesh.mandhare Mahesh Mandhare (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty