Resolution: Not a Bug
- Create a 3 nodes colocated services (k:q:i:a) cluster, a bucket, 2 collections. Load some data in collections. Build indexes, create datasets etc.
- Start n1ql/cbas query workload
- Scale up the cluster from 3 to 6 nodes, 6 to 9 nodes. Everything went fine.
- Scale down the cluster from 9 to 6 nodes and nodes started removing from the cluster 1 at a time.
- While node removal is nearing completion some cbas/n1ql queries started hitting RequestCancelledExceptions based on which service is rebalancing and traffic is switching.
- It looks like a race between cluster map shared to the client and client has already dispatched few requests to the outgoing node.
First instance of Exception for CBAS:
This is seen during:
Rebalance order by service on a node: KV -> FTS -> Index -> Analytics -> N1ql
cc: Ritam Sharma