Uploaded image for project: 'Java DCP Client'
  1. Java DCP Client
  2. JDCP-56

DCP client is missing updates on docs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.16.0
    • Labels:
      None
    • Environment:
      Our system test environment consists of many microservices deployed as docker containers. The storage is provided by couchbase version 4.6, also deployed as a docker container, no replication. All the containers are deployed on a single Linux Centos7 VM.

      Description

      During our system tests we make mutations on docs - sometimes updates do not arrive to client. It's not related to specific test or doc, every time something else don't arrive to client.

      Between tests we do not reinstall couchbase, only delete docs.

      Attaching createDcpClient code and some logs from last failure:

      1. WARNINGS on DCP from couchbase logs
      2. Client logs with exceptions - after this stack trace data started to arrive to listener but some date was lost.

       

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            ingenthr Matt Ingenthron added a comment -

            Do note Avital Brandwine that DCP does not guarantee you will see all mutations. Within a snapshot, DCP may deduplicate mutations to efficiently use system resources. It is not the same as a transaction log, but rather a stream of changes with guarantees about seeing the most recent copy of a given item.

            Sergey Avseyev may have more thoughts when he gets a chance to review.

            Show
            ingenthr Matt Ingenthron added a comment - Do note Avital Brandwine that DCP does not guarantee you will see all mutations. Within a snapshot, DCP may deduplicate mutations to efficiently use system resources. It is not the same as a transaction log, but rather a stream of changes with guarantees about seeing the most recent copy of a given item. Sergey Avseyev may have more thoughts when he gets a chance to review.
            Hide
            abrandwi Avital Brandwine (Inactive) added a comment -

            Thanks, but in our case we missed updates of new documents - so there was nothing to deduplicate. And we saw exceptions in logs.

            Show
            abrandwi Avital Brandwine (Inactive) added a comment - Thanks, but in our case we missed updates of new documents - so there was nothing to deduplicate. And we saw exceptions in logs.
            Hide
            avsej Sergey Avseyev added a comment -

            It turns out, that fix requires improvements on how we handle rollbacks. Currently JDCP does not allow to reopen streams from the control handler clearly, so when the user receives rollback event, he cannot react on it. I will assign this ticket to the next release and we are working on the fix.

            Show
            avsej Sergey Avseyev added a comment - It turns out, that fix requires improvements on how we handle rollbacks. Currently JDCP does not allow to reopen streams from the control handler clearly, so when the user receives rollback event, he cannot react on it. I will assign this ticket to the next release and we are working on the fix.
            Hide
            ingenthr Matt Ingenthron added a comment -

            Sergey Avseyev and David Nault: I think this issue was probably logically solved with rollback mitigation. do you agree?

            Show
            ingenthr Matt Ingenthron added a comment - Sergey Avseyev and David Nault : I think this issue was probably logically solved with rollback mitigation. do you agree?
            Hide
            avsej Sergey Avseyev added a comment -

            I agree, that it should be fixed by David Nault quite some time ago.

            Show
            avsej Sergey Avseyev added a comment - I agree, that it should be fixed by David Nault quite some time ago.
            Hide
            david.nault David Nault added a comment -

            Agreed, rollback mitigation should isolate the user's code from any checkpoint hiccups.

            Incidentally, the warnings in the logs about the "open stream" failures indicate a transient error condition; the "open stream" operation is retried indefinitely until it succeeds.

            Recommendation is to upgrade to the latest DCP client release (0.23.0 at time of writing) and enable rollback mitigation. If the problem persists, we'll need to look at the `updateDcpCheckPoint` method to validate that the expectations in the user code are correct.

            Show
            david.nault David Nault added a comment - Agreed, rollback mitigation should isolate the user's code from any checkpoint hiccups. Incidentally, the warnings in the logs about the "open stream" failures indicate a transient error condition; the "open stream" operation is retried indefinitely until it succeeds. Recommendation is to upgrade to the latest DCP client release (0.23.0 at time of writing) and enable rollback mitigation. If the problem persists, we'll need to look at the `updateDcpCheckPoint` method to validate that the expectations in the user code are correct.
            Hide
            david.nault David Nault added a comment -

            Marking this as fixed in 0.16.0 (when the rollback mitigation feature was first added). Subsequent releases improved on the robustness of rollback mitigation; any testing should be done against the latest version (0.23.0 at time of writing).

            Show
            david.nault David Nault added a comment - Marking this as fixed in 0.16.0 (when the rollback mitigation feature was first added). Subsequent releases improved on the robustness of rollback mitigation; any testing should be done against the latest version (0.23.0 at time of writing).

              People

              Assignee:
              david.nault David Nault
              Reporter:
              abrandwi Avital Brandwine (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty