[BP MB-51743 7.1.1] - Eventing function should be able to listen to multiple collections in a bucket at the same time

Description

In deployments with 100s of collections per bucket and 1 eventing function per collection, eventing ends up spawning at least 1 process per function and 100s of DCP connections with KV nodes. This can put resources on KV and Eventing nodes under undue stress.

Eventing in Neo supports full RBAC. So it should be possible for a single eventing function to listen to multiple collections in the same bkt at the same time provided user has necessary DCP data read permissions. One possible way to do this is as below

Option-1 (easiest):
Listen to Keyspace
1. bkt.scope.collection <----- as of Neo, function can listen to a single collection
2. bkt.scope.{*} <---- proposed option to listen to all collections in scope 'bkt.scope'
3. bkt.*.* <---- proposed option to listen to all collections in the bucket 'bkt'

Option-2 (a bit complicated):
1. Allow selection of multiple collections in the drop down 'Listen to Keyspace'. We may have to limit to a single bucket though and listening to collections across bkts can be problematic and non-intuitive.

Cc

Components

Affects versions

Fix versions

Labels

Environment

None

Release Notes Description

None

Activity

Show:

CB robot October 7, 2022 at 8:54 AM

Build couchbase-server-7.2.0-5000 contains eventing commit ba8c37d with commit message:
: Listen to multiple collection

CB robot October 7, 2022 at 8:53 AM

Build couchbase-server-7.2.0-5000 contains eventing commit e9024a6 with commit message:
: Include wildcard entry in recursion check

CB robot October 7, 2022 at 8:53 AM

Build couchbase-server-7.2.0-5000 contains eventing commit cf62856 with commit message:
: Fix source bucket mutation detection

CB robot October 7, 2022 at 8:53 AM

Build couchbase-server-7.2.0-5000 contains eventing commit 7526570 with commit message:
: Filter collection document when collection gets deleted

Sujay Gad July 11, 2022 at 5:40 AM

Closing this ticket as testing for multi collection feature has been completed.
Set of P0 functional tests have been added by dev team as part of CI - .
Component level system as well as volume test scripts have been modified to incorporate multi collection feature.
Additionally, 15 topology tests have been added in which eventing functions are listening at either bucket or scope level.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created May 30, 2022 at 4:02 AM
Updated August 31, 2024 at 11:06 AM
Resolved May 30, 2022 at 6:12 AM
Instabug