Details
Description
Steps to reproduce -
- Have a 4 node cluster as follows -
Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
10.112.205.102 | index, kv, n1ql | 54.1622760801 | 1.80 GiB | 1.16 GiB | 0.0 Byte / 1.03 GiB | 5000 / 5000 | 7.0.3-7031-enterprise |
10.112.205.104 | cbas | 55.1369863014 | 1.80 GiB | 919.89 MiB | 0.0 Byte / 1.03 GiB | 0 / 0 | 7.0.3-7031-enterprise |
10.112.205.103 | cbas | 1.73116089613 | 1.80 GiB | 784.16 MiB | 126.06 MiB / 1.03 GiB | 0 / 0 | 7.0.3-7031-enterprise |
10.112.205.101 | index, kv, n1ql | 70.3974895397 | 1.80 GiB | 1.19 GiB | 121.04 MiB / 1.03 GiB | 5000 / 5000 | 7.0.3-7031-enterprise |
2. create a bucket and load some data into it.
3. Create dataverses, datasets and indexes in cbas.
4. let the ingestion into datasets complete.
5. Gracefully failover one of the kv node.
6. Upgrade the failed over node to 7.1.0-2215 build.
7. Set recovery type as delta for failed over node and add it back and rebalance. Cluster would look like below
Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
10.112.205.102 | index, kv, n1ql | 54.1622760801 | 1.80 GiB | 1.16 GiB | 0.0 Byte / 1.03 GiB | 5000 / 5000 | 7.0.3-7031-enterprise |
10.112.205.104 | cbas | 55.1369863014 | 1.80 GiB | 919.89 MiB | 0.0 Byte / 1.03 GiB | 0 / 0 | 7.0.3-7031-enterprise |
10.112.205.103 | cbas | 1.73116089613 | 1.80 GiB | 784.16 MiB | 126.06 MiB / 1.03 GiB | 0 / 0 | 7.0.3-7031-enterprise |
10.112.205.101 | index, kv, n1ql | 70.3974895397 | 1.80 GiB | 1.19 GiB | 121.04 MiB / 1.03 GiB | 5000 / 5000 | 7.1.0-2215-enterprise |
8. Once the rebalance completes, try accessing analytics.
Following error is seen -
Received error 503 when contacting the analytics service. Try refreshing the browser, as errors may occur if the analytics service is still warming up or suffering a transient error.
9. Follow steps 5-8 again for the other KV node. Same error is seen for analytics.
Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version |
10.112.205.102 | index, kv, n1ql | 54.1622760801 | 1.80 GiB | 1.16 GiB | 0.0 Byte / 1.03 GiB | 5000 / 5000 | 7.1.0-2215-enterprise |
10.112.205.104 | cbas | 55.1369863014 | 1.80 GiB | 919.89 MiB | 0.0 Byte / 1.03 GiB | 0 / 0 | 7.0.3-7031-enterprise |
10.112.205.103 | cbas | 1.73116089613 | 1.80 GiB | 784.16 MiB | 126.06 MiB / 1.03 GiB | 0 / 0 | 7.0.3-7031-enterprise |
10.112.205.101 | index, kv, n1ql | 70.3974895397 | 1.80 GiB | 1.19 GiB | 121.04 MiB / 1.03 GiB | 5000 / 5000 | 7.1.0-2215-enterprise |