Uploaded image for project: 'Couchbase Documentation'
  1. Couchbase Documentation
  2. DOC-9146

Eventing N2N encryption: Add note on precautions while changing encryption levels

    XMLWordPrintable

Details

    • Task
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 7.0.2
    • None
    • eventing
    • None
    • 1

    Description

      It is very important that in N2N encryption docs, for Eventing we ensure that the users are aware of the following scenarios:

      Let's take a javascript function that runs a set of bucket ops followed by UPSERT INTO DML queries later on. This function is currently executing and has successfully pushed KV ops but execution hasn't reached the DML query yet.
      EnforceTLS notification comes mid-way which causes the libcouchbase handles to go stale causing query failure. This will be reported as an on_update_failure, however KV requests have already been pushed. In other words, eventing functions are almost always non-idempotent and hence cannot be retried.
      This mutation is not lost but rather "half lost" which is still not good as far as customer's production environment is concerned.

      As such a use case holds true for Eventing and not for any other indexing service like analytics, FTS, GSI we need to highlight that customers either pause their handlers / or do not push any traffic to src bucket (pausing is necessary if mutation src is timer store) prior to changing encryption levels (especially when changing to strict).

      On similar lines, due to cbauth callbacks being eventually consistent in nature, if an eventing function is processing mutations at X mutations per second, and callback gets called only after 1 second, user can loose those X mutations.

      Additionally:

      • Once encryption level has been changed from control -> all/ strict or from all/strict -> control / disabled , it is important that users either pause-resume or undeploy-redeploy all currently deployed eventing handlers to ensure all connections are running at the appropriate encryption level.
      • Once encryption level has been changed from control -> all/ strict or from all/strict -> control / disabled, subsequent eventing rebalances can get stuck or fail unless handlers have been pause-resumed or undeploy / redeployed.
      • We recommend users to not change encryption level while eventing handlers are undergoing lifecycle operations.
      • We recommend users to not change encryption level while eventing rebalance is ongoing.

      If either deployment / rebalance fails or gets stuck due to any of the above scenarios, as a workaround the best possible option would be to set the encryption level back to the previous value which will help in getting out a failed / stuck rebalance or lifecycle operation.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jeelan.poola Jeelan Poola added a comment - Cc Jon Strabala Ian McCloy
          jeelan.poola Jeelan Poola added a comment -

          In summary, Users should pause eventing functions before changing encryption settings and resume soon after to avoid losing or running into partial mutation processing.

          This would also be true for any application traffic originating from outside of couchbase cluster. SDK operations can fail mid-way if encryption settings are changed and application is connected over plain text.

          jeelan.poola Jeelan Poola added a comment - In summary, Users should pause eventing functions before changing encryption settings and resume soon after to avoid losing or running into partial mutation processing . This would also be true for any application traffic originating from outside of couchbase cluster. SDK operations can fail mid-way if encryption settings are changed and application is connected over plain text.

          People

            jon.strabala Jon Strabala
            abhishek.jindal Abhishek Jindal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty