Details
Critical Description
Steps to reproduce -
- Have a 6 node cluster as follows -
Nodes Services Version CPU Status
| 172.23.104.217 | kv | 7.1.0-2250-enterprise | 1.86351045077 | Cluster node |
| 172.23.104.163 | cbas | 7.1.0-2250-enterprise | 8.14274943453 | Cluster node |
| 172.23.104.201 | cbas | 7.1.0-2250-enterprise | 24.4670050761 | Cluster node |
| 172.23.104.202 | cbas | 7.1.0-2250-enterprise | 38.2149269043 | Cluster node |
| 172.23.104.161 | index, kv, n1ql | 7.1.0-2250-enterprise | 4.44556706239 | Cluster node |
| 172.23.104.179 | cbas | 7.1.0-2250-enterprise | 9.70677451972 | Cluster node |
2. Set analytics replica as 3.
3. Create KV side infra (buckets, scopes and collections) and load data.
4. Create CBAS side infra (dataverses, datasets, indexes, synonyms etc)
5. Rebalance out any 2 cbas nodes.
6. While rebalance is going on, stop couchbase server on one of the outgoing node.
7. Rebalance fails as expected.
8. Now restart the server that was stopped in step 6.
9. Rebalance out any 2 cbas nodes.
Rebalance fails.
This issue was not caught in previous build due to automation issue, once that was fixed we hit this issue.
Also the issue with rebalance is intermittent, sometime it passes without a glitch.