Details
Description
Following error is observed:
2023-07-25T01:24:07.441-07:00 WARN CBAS.cbas got error obtaining current creds from cbauth: Unable to find given hostport in cbauth database: `172.23.120.90:11207'; will retry |
2023-07-25T01:24:08.442-07:00 WARN CBAS.cbas got error obtaining current creds from cbauth: Unable to find given hostport in cbauth database: `172.23.120.90:11207'; will retry |
Steps to reproduce:
- Create a 6 node cluster with 3 KV and 3 CBAS node.
- Create 2 buckets on KV, with each bucket containing 3 scopes and 2 collection per scope.
- Load 10K docs in each collection except the default collection in each scope.
- Set analytics replica to 2
- create 2 analytics scopes (default included) and 2 analytics collection in each scope.
- Let the data be ingested into datasets.
- Rebalance in a KV node.
- Start parallel crud operation on each collection with deleting 250 docs, setting 10 sec expiry for another 250 docs, updating another 250 docs and creating 500 new docs.
- Once rebalance completes, validate doc count in each dataset. Doc count check passed.
- Rebalance in CBAS node on the cluster with Loading of docs on KV
- Wait for rebalance to complete
- Rebalance out 1 CBAS node and 1 KV node on the cluster with Loading of docs
- Wait for rebalance to complete
- Rebalance In a CBAS node and a data node on the cluster with Loading of docs
- Wait for rebalance to complete
- Rebalance Out data node with Loading of docs
- Wait for rebalance to complete
- Rebalance Out CBAS node with Loading of docs on KV
- Wait for rebalance to complete
- Rebalance In-Out data node with Loading of docs.
- Wait for rebalance to complete
- Rebalance In-Out (2 in, 1 out) CBAS node on the cluster with Loading of docs on KV(please see attachment for image)
- Start parallel crud operation on each collection with deleting 250 docs, setting 10 sec expiry for another 250 docs, updating another 250 docs and creating 500 new docs.
The rebalance is stuck after the above steps.