[BP MB-39877] - eventing crashes due to a race between undeploy and delete

Ankit Prabhu diagnosed the issue on Slack

So its a race condition between undeployment and delete. When we click on delete in ui it will 1st delete the tempstore and then proceed to delete from primary store.

In primary store it will delete the settings path first
and then check whether the app is undergoing undeployment or not
https://github.com/couchbase/eventing/blob/5d15449549fcc7770fa1facdbddbbf063c0b5fb4/service_manager/http_handlers.go#L135

service_manager/http_handlers.go:135

    err := util.MetaKvDelete(settingPath, nil)

https://github.com/couchbase/eventing/blob/5d15449549fcc7770fa1facdbddbbf063c0b5fb4/service_manager/http_handlers.go#L146

service_manager/http_handlers.go:146

        info.Info = fmt.Sprintf("Function: %s is currently undergoing undeploy. Only undeployed function can be deleted", appName)

Now since its undergoing undeployment it will simply return and it won't delete the primary store.
Now those checksome and definition remains in primary store
When we rebalance other nodes then it will try to read the settings from setting path and it won't be able to read and the interface will be nil

2020-06-07T09:35:28.007-05:00 [Error] ServiceMgr::parseFunctionPayload failed to unmarshal settings data from metakv, err: unexpected end of JSON input

It will exit it by giving the interface conversion error.