Details
-
Bug
-
Resolution: Unresolved
-
Major
-
6.5.0
-
Untriaged
-
Unknown
Description
I was doing some normal handler deploy/undeploy/editing/deploy cycles (slowly and carefully, not a stress test) and the function got stuck in Undeploying state.
curl 'http://click:8096/api/v1/status'
|
{
|
"apps": [
|
{
|
"composite_status": "undeploying",
|
"name": "test",
|
"num_bootstrapping_nodes": 0,
|
"num_deployed_nodes": 1,
|
"deployment_status": false,
|
"processing_status": false
|
}
|
],
|
"num_eventing_nodes": 1
|
}
|
The cluster is quite healthy. All buckets are online and UI works fine. I can even export the handler without trouble from UI after it's stuck in this state. See eventing.log
Then at 2019-07-19T18:09:31.330 I did "sudo killall eventing-producer eventing-consumer". The processes restarted, but it is still stuck undeploying. See eventing2.log
Out of desperation, I did "sudo killall -9 eventing-producer eventing-consumer" at 2019-07-19T18:15:58.577 and it restarted again (verified by observing PIDs of eventing-consumer and eventing-producer) change. It still is undeploying!
At this point, I'm giving up and attaching [^collectinfo-2019-07-19T125046-ns_1@127.0.0.1.zip]
Note that I did curl 'http://click:8096/api/v1/functions/test' and it shows:
{
|
"appcode": "function OnUpdate(doc, meta) {\n var r = dst['idx'];\n if (!r || !r.n) r = {'n': 0};\n r.n++;\n dst['idx'] = r;\n var r = Math.floor(Math.random() * 6);\n var m = {\"hello_m\": \"world_m\"};\n var a = [\"hello_a\", \"world_a\"];\n var i = (r.n) % 7;\n log(\"Starting run:\", i);\n switch (i) {\n case 0: throw \"foo1\";\n case 1: no_such_method();\n case 2: bad_var++;\n case 3: throw m;\n case 4: throw a;\n case 5: throw 123.45;\n case 6: throw true;\n }\n log(\"Finished run:\", i);\n}",
|
"depcfg": {
|
"buckets": [
|
{
|
"alias": "dst",
|
"bucket_name": "dst",
|
"access": "rw"
|
}
|
],
|
"curl": [],
|
"metadata_bucket": "meta",
|
"source_bucket": "src"
|
},
|
"version": "evt-6.5.0-0000-ee",
|
"function_id": 3048009367,
|
"id": 0,
|
"function_instance_id": "E5SjJ3",
|
"appname": "test",
|
"settings": {
|
"dcp_stream_boundary": "everything",
|
"deadline_timeout": 62,
|
"deployment_status": false,
|
"description": "",
|
"execution_timeout": 60,
|
"log_level": "INFO",
|
"processing_status": false,
|
"user_prefix": "eventing",
|
"using_timer": false,
|
"worker_count": 1
|
},
|
"using_timer": false,
|
"src_mutation": false
|
}
|
*Note:
"deployment_status": false,
"processing_status": false*
But REST API says:
{
|
"apps": [
|
{
|
"composite_status": "undeploying",
|
"name": "test",
|
"num_bootstrapping_nodes": 0,
|
"num_deployed_nodes": 1,
|
"deployment_status": false,
|
"processing_status": false
|
}
|
],
|
"num_eventing_nodes": 1
|
}
|
Attachments
Issue Links
- backports to
-
-
- Closed
-
Gerrit Reviews
| For Gerrit Dashboard: MB-35188 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 112314,2 | MB-35188: Do not write to metakvappsettings path at the end of deployment | unstable | eventing | Status: MERGED | +2 | +1 |
| 113011,3 | MB-35188 MB-35462: Add retries around all metakv writes | unstable | eventing | Status: MERGED | +2 | +1 |