Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
None
-
None
-
Untriaged
-
Centos 64-bit
-
Unknown
Description
We have registered for changes in the metakv path /eventing/rebalanceToken/ to get notifications about rebalance using metakv.RunObserveChildren API - https://github.com/couchbase/eventing/blob/master/cmd/producer/main.go#L65
We log the path and the token that is passed by metakv whenever the callback is invoked - https://github.com/couchbase/eventing/blob/master/supervisor/super_supervisor.go#L387
In one of our CI runs - http://ci-eventing-rebalance.northscale.in/eventing-30.03.2019-08.09.fail.html , we observed in the eventing logs that the callback was getting invoked spuriously with the same token and path.
$ zgrep "SuperSupervisor::TopologyChangeNotifCallback \[1\] Path =>" eventing.log* | wc -l |
491 |
|
$ zgrep "SuperSupervisor::TopologyChangeNotifCallback \[1\] Path => /eventing/rebalanceToken/af80d2bf8f45b5a7fa6513d41d349bbd value => af80d2bf8f45b5a7fa6513d41d349bbd" eventing.log* | wc -l |
491 |
The spurious invocations are causing rebalance to be triggered in eventing, causing the Functions to get stuck. From what I understand, the callback must be called only when the topology changes.
Logs s3://cb-customers-secure/couchbase/2019-04-02/logs-30.03.2019-08.09.tar.gz
After unzipping, goto to the directory logs-functional-ns_server/logs/
Attachments
Issue Links
- causes
-
MB-33084 Deploy/Undeploy of a handler getting stuck intermittently
- Closed