Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-57997

Rebalance Failure while removing one CBAS node and adding 2 CBAS nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • Morpheus
    • Morpheus
    • analytics
    • Enterprise Edition 8.0.0 build 1355

    Description

      Following error is observed:

      2023-07-25T01:24:07.441-07:00 WARN CBAS.cbas got error obtaining current creds from cbauth: Unable to find given hostport in cbauth database: `172.23.120.90:11207'; will retry
      2023-07-25T01:24:08.442-07:00 WARN CBAS.cbas got error obtaining current creds from cbauth: Unable to find given hostport in cbauth database: `172.23.120.90:11207'; will retry
      

      Steps to reproduce:

      1. Create a 6 node cluster with 3 KV and 3 CBAS node.
      2. Create 2 buckets on KV, with each bucket containing 3 scopes and 2 collection per scope.
      3. Load 10K docs in each collection except the default collection in each scope.
      4. Set analytics replica to 2
      5. create 2 analytics scopes (default included) and 2 analytics collection in each scope.
      6. Let the data be ingested into datasets.
      7. Rebalance in a KV node.
      8. Start parallel crud operation on each collection with deleting 250 docs, setting 10 sec expiry for another 250 docs, updating another 250 docs and creating 500 new docs.
      9. Once rebalance completes, validate doc count in each dataset. Doc count check passed.
      10. Rebalance in CBAS node on the cluster with Loading of docs on KV
      11. Wait for rebalance to complete
      12. Rebalance out 1 CBAS node and 1 KV node on the cluster with Loading of docs
      13. Wait for rebalance to complete
      14. Rebalance In a CBAS node and a data node on the cluster  with Loading of docs
      15. Wait for rebalance to complete
      16. Rebalance Out data node with Loading of docs
      17. Wait for rebalance to complete
      18. Rebalance Out CBAS node with Loading of docs on KV
      19. Wait for rebalance to complete
      20. Rebalance In-Out data node with Loading of docs.
      21. Wait for rebalance to complete
      22. Rebalance In-Out (2 in, 1 out) CBAS node on the cluster with Loading of docs on KV(please see attachment for image)
      23. Start parallel crud operation on each collection with deleting 250 docs, setting 10 sec expiry for another 250 docs, updating another 250 docs and creating 500 new docs.

      The rebalance is stuck after the above steps.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              abhay.aggrawal Abhay Aggrawal
              abhay.aggrawal Abhay Aggrawal
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty