We need a metadata store that is immediately consistent.
The metadata store should begin with nodes, buckets and collections and should eventually include things such as GSI indexes. An example use case for indexing that fails today that this would fix would be as follows:
[GSI] Indexes can be created with duplicate index_names when one of the indexer node is in failed over state
1. Configure cluster. Create bucket default and load the bucket with 10k items.
2. Create index ind_3 on default(name). It gets created on node 2.
3. Failover Node 2.
4. Create index ind_3 on default(age). It gets created on node 1.
5. Rebalance in Node 2 with full recovery.
Now there are 2 indexes with same name "ind_3" with different index definitions.
when dropping ind_3, later one gets dropped. There is no way of controlling drop behaviour in such cases.
Expectation is, indexes with duplicate index_names shouldn't be allowed to be created.