Need document routing to support join

Description

This version does not support document routing to support joins. Currently to be able to do this, I have to use the ElasticSearch API to place the Child Docs into the Index, rather than create Couchbase Documents and push them through the Connector

Environment

None

Gerrit Reviews

None

Release Notes Description

None

Activity

Show:

Keith Lubell May 1, 2019 at 4:43 AM

Awesome. I see 4.0.1 is out. I will give it a go

David Nault March 11, 2019 at 9:54 PM

Partial solution (without deletion) will be included in next release.

Keith Lubell February 8, 2019 at 6:56 PM
Edited

In my use case, the parent is usually in the bucket long before the children. And usually the parent is never deleted. A simple route would do for this.

As I noted in GitHub "I would have loved this feature too. I got around the problem by writing the child documents directly to ElasticSearch using their API client. I know the ElasticSearch people are not keen on Joins either, but it is hard denormalize a relationship where there is one parent for thousands of children. In an interesting note. The ES API has some nice features to do bulk indexing and deletes".

I do know the old connector had a problem sometime writing the child docs before the parent docs causing issues. This only came when I had to rebuild the index from scratch and direct from Couchbase.

 

David Nault February 7, 2019 at 2:13 AM

Because the DCP protocol used by the connector does not include document bodies in deletion notifications, it's not easy to know where to route the deletion requests.

See discussion over at https://github.com/couchbase/couchbase-elasticsearch-connector/pull/203 where one of the proposals was to support routing for a document type only when the connector is configured to ignore deletions for that type.

Another caveat is that because each virtual bucket is replicated independently, in general there's no guarantee that two different documents will be indexed in Elasticsearch in the same order they were written to Couchbase (child might be indexed before parent if they are in different vbuckets, for example).
I haven't experimented with parent/join enough to know whether that's likely to be a problem.

@envitraux If you could specify a routing but not delete the routed documents, and the parent and children could be indexed in any order, would that satisfy your use case?

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Fix versions

Affects versions

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created February 6, 2019 at 3:03 PM
Updated May 1, 2019 at 4:43 AM
Resolved March 11, 2019 at 9:54 PM
Instabug