Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
6.5.0
-
Untriaged
-
Yes
Description
Build 6.5.0-4926
Observed eventing worker died message while doing kv node out rebalance in high bucket density test, due to which rebalance failed.
Following eventing function deployed in cluster-
{
|
"actionsVisible": true,
|
"appcode": "function OnUpdate(doc, meta) {\n bucket1[meta.id]=doc[\"alt_email\"];\n}\n\nfunction OnDelete(doc) {\n}\n",
|
"appname": "bucket-1-test1",
|
"depcfg": {
|
"buckets": [
|
{
|
"alias": "bucket1",
|
"bucket_name": "eventing-bucket-1"
|
}
|
],
|
"metadata_bucket": "eventing",
|
"source_bucket": "bucket-1"
|
},
|
"settings": {
|
"breakpad_on": false,
|
"cpp_worker_thread_count": 2,
|
"dcp_stream_boundary": "from_now",
|
"deployment_status": true,
|
"description": "Perf test",
|
"enable_recursive_mutation": true,
|
"log_level": "INFO",
|
"processing_status": true,
|
"timer_worker_pool_size": 1,
|
"worker_count": 3,
|
"worker_queue_cap": 100000
|
}
|
}
|
Rebalance failure message-
"Rebalance exited with reason {service_rebalance_failed,eventing
|
,\n {worker_died,\n {'EXIT',<0.20276.305>,\n {rebalance_failed,\n
|
\{service_error,\n <<\"eventing rebalance hasn't made progress for past 600 secs\">>}}}}}."}
|
Job- http://perf.jenkins.couchbase.com/job/arke-multi-bucket/339
Logs-
Eventing node- https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.177.zip
Other nodes-
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.16.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.17.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.20.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.96.23.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.12.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.13.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.14.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.15.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.19.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-arke-multi-bucket-339/172.23.97.20.zip
We did not observed this issue in 6.5.0-4908.