Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30772

Killing eventing processes can cause timers in flight to be lost

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 6.0.0
    • Fix Version/s: Mad-Hatter
    • Component/s: eventing
    • Labels:
    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Is this a Regression?:
      Unknown

      Description

      It's a known limitation that if eventing process is killed externally, it may lose timers that have been dequeued but not run to completion. This is because eventing-producer does not follow a Request & Acknowledge protocol when queuing events to v8 workers. As a result, forcibly killing the eventing producer OS process will cause the timers that have been dequeued from storage but not yet fired by v8 workers to be lost.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            siri Sriram Melkote added a comment -

            The solution to this is to have a processing request & acknowledge handshake, to ensure that v8 worker completion of timer events is tracked. This is WIP, tracked by this bug.

            Show
            siri Sriram Melkote added a comment - The solution to this is to have a processing request & acknowledge handshake, to ensure that v8 worker completion of timer events is tracked. This is WIP, tracked by this bug.
            Hide
            venkatraman.subramanian Venkatraman Subramanian added a comment -

            Is this behaviour only for Timers or even for generic event processing(i.e, normal handlers)?

            Show
            venkatraman.subramanian Venkatraman Subramanian added a comment - Is this behaviour only for Timers or even for generic event processing(i.e, normal handlers)?
            Hide
            Gautham.Banasandra Gautham Banasandra added a comment -

            Venkatraman Subramanian This behaviour is specific to Timers only. We'll need to build a request-ack protocol, which guarantees that the timer has fired before deleting it.

            Show
            Gautham.Banasandra Gautham Banasandra added a comment - Venkatraman Subramanian This behaviour is specific to Timers only. We'll need to build a request-ack protocol, which guarantees that the timer has fired before deleting it.
            Hide
            venkatraman.subramanian Venkatraman Subramanian added a comment -

            We need the req-ack across the board for better reliability. MB-29083 was filed to track this.

            Show
            venkatraman.subramanian Venkatraman Subramanian added a comment - We need the req-ack across the board for better reliability.  MB-29083  was filed to track this.
            Hide
            siri Sriram Melkote added a comment -

            Gautham Banasandra I've committed a new API to get a portable reference and updated the implementation to work with external deletions. You can call GetToken() on an entry, and pass that along to C++ and delete the keys using the provided CAS safely from C++ without concern about other parts of the structure. See DeleteToken for documentation.

            Show
            siri Sriram Melkote added a comment - Gautham Banasandra I've committed a new API to get a portable reference and updated the implementation to work with external deletions. You can call GetToken() on an entry, and pass that along to C++ and delete the keys using the provided CAS safely from C++ without concern about other parts of the structure. See DeleteToken for documentation.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.0.0-1605 contains eventing commit da0b064 with commit message:
            MB-30772: Allow deleting timer from outside Go

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.0-1605 contains eventing commit da0b064 with commit message: MB-30772 : Allow deleting timer from outside Go
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.5.0-1287 contains eventing commit da0b064 with commit message:
            MB-30772: Allow deleting timer from outside Go

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.0-1287 contains eventing commit da0b064 with commit message: MB-30772 : Allow deleting timer from outside Go

              People

              • Assignee:
                satya.nand Satya Nand
                Reporter:
                Balakumaran.Gopal Balakumaran Gopal
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Gerrit Reviews

                  There are no open Gerrit changes

                    PagerDuty

                    Error rendering 'com.pagerduty.jira-server-plugin:PagerDuty'. Please contact your Jira administrators.