Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33601

Spurious callback invocation on /eventing/rebalanceToken/ metakv path

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 6.5.0
    • None
    • eventing, ns_server
    • None

    Description

      We have registered for changes in the metakv path /eventing/rebalanceToken/ to get notifications about rebalance using metakv.RunObserveChildren API - https://github.com/couchbase/eventing/blob/master/cmd/producer/main.go#L65

      We log the path and the token that is passed by metakv whenever the callback is invoked - https://github.com/couchbase/eventing/blob/master/supervisor/super_supervisor.go#L387

      In one of our CI runs - http://ci-eventing-rebalance.northscale.in/eventing-30.03.2019-08.09.fail.html , we observed in the eventing logs that the callback was getting invoked spuriously with the same token and path.

      $ zgrep "SuperSupervisor::TopologyChangeNotifCallback \[1\] Path =>" eventing.log* | wc -l
          491
       
      $ zgrep "SuperSupervisor::TopologyChangeNotifCallback \[1\] Path => /eventing/rebalanceToken/af80d2bf8f45b5a7fa6513d41d349bbd value => af80d2bf8f45b5a7fa6513d41d349bbd" eventing.log* | wc -l
          491 

      The spurious invocations are causing rebalance to be triggered in eventing, causing the Functions to get stuck. From what I understand, the callback must be called only when the topology changes.

      Logs  s3://cb-customers-secure/couchbase/2019-04-02/logs-30.03.2019-08.09.tar.gz
      After unzipping, goto to the directory logs-functional-ns_server/logs/

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Gautham.Banasandra Gautham Banasandra (Inactive)
              Gautham.Banasandra Gautham Banasandra (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty