Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31625 [FTS] Support for collections
  3. MB-36479

[gocbcore] DCP Connection sharing amongst multiple FTS indexes with the same source bucket

    XMLWordPrintable

Details

    • Technical task
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • None
    • fts

    Description

      Supporting several FTS indexes on a couchbase cluster (in other words FTS multi tenancy) has one critical short-coming with the current design - each FTS index creates the "index_partition_count * (num_replicas + 1)" number of DCP connections. The system would bottle neck on the number of such TCP connections created with the growing number of indexes at some point. 

      Here are proposed steps to solving this ..

      • A form of connection sharing across all indexes with the same source collection/bucket for a given node.
      • Owing to the index set up times, new streams can be instantiated either within the existing connection or with a new connection for that index. Once the index has caught up with the other index(es), these multiple streams/connections would need to be consolidated to a single stream/connection that would cater to all the indexes (streaming from the same source).

      Consolidation (or merging) in the step above could simply mean dropping all the existing streams/connections and recreating one new stream/connection.

      Each stream’s SnapshotMarker/Mutation/Deletion/Expiration callbacks would now be invoked for all the FTS dests to update each of the associated indexes.

      Ideally speaking, it’d be awesome if the SDK would support this connection/stream merging capability. So bringing this to the radar of Brett Lawson to get any thoughts and insights he’d have here for us.

      Attachments

        Issue Links

          Activity

            People

              abhinav Abhi Dangeti
              abhinav Abhi Dangeti
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty