Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57853

Cluster hung in rebalancing state

    XMLWordPrintable

Details

    • Bug
    • Resolution: Known Error
    • Critical
    • 7.2.1
    • 7.2.1
    • ns_server
    • Untriaged
    • 0
    • Unknown

    Description

      Provisioned - AWS - couchbase-cloud-server-7.2.1-5819-v1.0.20, m5.2xlarge

      Steps to reproduce: 

      Scenario 1.

      1. Create 3 magma buckets
      2. Create 4 scopes in each bucket and 5 collections in each scope
      3. Load 7M data in each bucket

      On doing this the cluster goes into an infinite rebalancing mode. 

      Logs:

      1. https://cb-engineering.s3.amazonaws.com/logs/reblancelogs/log1.zip
      2. https://cb-engineering.s3.amazonaws.com/logs/reblancelogs/logs2.zip
      3. https://cb-engineering.s3.amazonaws.com/logs/reblancelogs/log3.zip

       

      oid=6f10f821-fbb5-4954-a697-98ec18c71dd9
      pid=b65d6399-b2cf-4ec3-b979-cbc5c1bad4a0
      dbid=baf3bdbb-1686-4b11-a00b-fe344a3ea997

       

      Scenario 2.

      1. Create 3 magma buckets
      2. Create 4 scopes in each bucket and 5 collections in each scope
      3. Load 3M data in each bucket
      4. Create FTS Indexes and run fts queries
      5. Kill and terminate process and nodes
      6. Scale out from 3 node cluster to 5 node cluster
      7. During this process there was another idle cluster which went into rebalance mode for no reason and hasn't recovered since.

      Logs:

      1. https://cb-engineering.s3.amazonaws.com/logs/MB-NODE1/newlogs.zip
      2. https://cb-engineering.s3.amazonaws.com/logs/MB-NODE1/newlogs2.zip
      3. https://cb-engineering.s3.amazonaws.com/logs/MB-NODE1/newlogs3.zip

      oid=6f10f821-fbb5-4954-a697-98ec18c71dd9
      pid=b65d6399-b2cf-4ec3-b979-cbc5c1bad4a0
      dbid=764440a9-405b-432d-9545-b2e5790deb04 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sarthak.dua Sarthak Dua
            sarthak.dua Sarthak Dua
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty