Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31625 [FTS] Support for collections
  3. MB-36479

[gocbcore] DCP Connection sharing amongst multiple FTS indexes with the same source bucket



    • Technical task
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • None
    • fts


      Supporting several FTS indexes on a couchbase cluster (in other words FTS multi tenancy) has one critical short-coming with the current design - each FTS index creates the "index_partition_count * (num_replicas + 1)" number of DCP connections. The system would bottle neck on the number of such TCP connections created with the growing number of indexes at some point. 

      Here are proposed steps to solving this ..

      • A form of connection sharing across all indexes with the same source collection/bucket for a given node.
      • Owing to the index set up times, new streams can be instantiated either within the existing connection or with a new connection for that index. Once the index has caught up with the other index(es), these multiple streams/connections would need to be consolidated to a single stream/connection that would cater to all the indexes (streaming from the same source).

      Consolidation (or merging) in the step above could simply mean dropping all the existing streams/connections and recreating one new stream/connection.

      Each stream’s SnapshotMarker/Mutation/Deletion/Expiration callbacks would now be invoked for all the FTS dests to update each of the associated indexes.

      Ideally speaking, it’d be awesome if the SDK would support this connection/stream merging capability. So bringing this to the radar of Brett Lawson to get any thoughts and insights he’d have here for us.


        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.



              abhinav Abhi Dangeti
              abhinav Abhi Dangeti
              0 Vote for this issue
              9 Start watching this issue



                Gerrit Reviews

                  There are no open Gerrit changes