Details
-
Epic
-
Resolution: Done
-
Major
-
Goldfish Private Preview
-
CBAS:Dynamic prefixes for external collections
-
1
-
CX Sprint 286, Analytics Sprint 1, Analytics Sprint 2, Analytics Sprint 3, Analytics Sprint 4, Analytics Sprint 5, Analytics Sprint 6
Description
Currently, when a user creates an external analytics collection, a prefix can be provided which directs the external collection to the location the files need to be read from. This has a major impact on performance as it allows us to only read the files we are interested in an avoid reading unnecessary files.
However, a limitation to the current implementation is that the prefix is always a static path, leading to challenges such as reading the file (for example) of all userId > 1 or all files of userId IN [1, 2, 3], in such scenarios we always end up reading all the files, which can be a very expensive operation, then using our WHERE clause to get the desired result.
This feature aims to support a more dynamic approach to allow for a flexible prefix that can support different scenarios (for example, the user passing the desired userId in the prefix instead of a single prefix value) and still maintain the behavior of reading the minimal number of files.