Details
-
Improvement
-
Resolution: Duplicate
-
Major
-
None
-
6.5.0
-
None
Description
The query engine needs to maintain an up-to-date picture of the set of collections and scopes in each bucket. These are listed, along with various UID version information, in the manifest of each bucket.
The KV engine provides us a way to get the manifest and notifies us when we attempt to access a collection that is no longer present. Unfortunately these are a bit clumsy for maintaining an up-to-date picture of each bucket. We would need to poll the KV engine periodically to check whether the manifest has changed. Also, when the manifest does change, we would either have to redo the entire data structure that lists the scopes and collection, or diff the old and new manifest to allow for more specific changes. Doing this would definitely work, but is likely to become problematic at scale, in a heavily used system with hundreds of scopes each containing hundreds of collections.
To that end we would like to be provided with a streaming interface that gives us more specific notification of collection and scope changes. Here are the features we would like to see:
- Streaming interface of notifications of manifest changes
- Per-bucket or per-cluster
- Suggested operations:
- Add scope
- Drop scope
- Add collection
- Drop collection
- Every notification contains original manifest UID and new manifest UID, so no changes are inadvertently skipped.
We are open to a number of ways of providing this, either by augmenting an existing stream or creating a new one. Or perhaps Meta-KV would be the right way to do it.
Attachments
Issue Links
- duplicates
-
MB-33236 notification of collections UID changes
- Closed