Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
6.6.0, 6.6.1
-
Untriaged
-
1
-
Unknown
Description
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.
- System should have more than 1 functions deployed on the same src-bkt
- KV rebalance should be triggered
- As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt
- If system hits this race condition, rebalance will be stuck and will fail after 10 mins.
customer deployment ran into this issue twice.
Attachments
Issue Links
- is a backport of
-
-
- Closed
-
Activity
| Link | This issue relates to CBSE-9247 [ CBSE-9247 ] |
| Fix Version/s | 6.6.1 [ 17002 ] | |
| Fix Version/s | Cheshire-Cat [ 15915 ] |
| Summary |
[BP of |
Undeploy/Pause closes source bkt reference even when there are other functions deployed against it |
| Affects Version/s | 6.6.1 [ 17002 ] |
| Assignee | Jeelan Poola [ jeelan.poola ] | Ankit Prabhu [ ankit.prabhu ] |
| Description |
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, many conditions have to be met to reproduce this problem.
* Below conditions have to be met to reproduce the problem * System should have more than 1 functions deployed on the same src-bkt * KV rebalance should be triggered * As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt * If system hits this race condition, rebalance will be stuck and will fail after 10 mins. customer deployment ran into this issue twice. |
| Description |
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, many conditions have to be met to reproduce this problem.
* Below conditions have to be met to reproduce the problem * System should have more than 1 functions deployed on the same src-bkt * KV rebalance should be triggered * As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt * If system hits this race condition, rebalance will be stuck and will fail after 10 mins. customer deployment ran into this issue twice. |
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.
* Below conditions have to be met to reproduce the problem * System should have more than 1 functions deployed on the same src-bkt * KV rebalance should be triggered * As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt * If system hits this race condition, rebalance will be stuck and will fail after 10 mins. customer deployment ran into this issue twice. |
| Description |
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.
* Below conditions have to be met to reproduce the problem * System should have more than 1 functions deployed on the same src-bkt * KV rebalance should be triggered * As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt * If system hits this race condition, rebalance will be stuck and will fail after 10 mins. customer deployment ran into this issue twice. |
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.
* Below conditions have to be met to reproduce the problem * System should have more than 1 functions deployed on the same src-bkt * KV rebalance should be triggered * As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt * If system hits this race condition, rebalance will be stuck and will fail after 10 mins. customer deployment ran into this issue twice. |
Jeelan Poola - RC1 for 6.6.1 is running. I propose we take this fix for next release for 6.6
| Description |
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.
* Below conditions have to be met to reproduce the problem * System should have more than 1 functions deployed on the same src-bkt * KV rebalance should be triggered * As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt * If system hits this race condition, rebalance will be stuck and will fail after 10 mins. customer deployment ran into this issue twice. |
This issue is reported by 6.6.0-MP1 customer. This causes eventing rebalance to get stuck and the only way out is to undeploy/pause the offending function. And figuring out the offending function would be difficult for customers, so it will force them to undeploy/pause all functions and redeploy/resume them to get out of the problem. However, below conditions have to be met to reproduce this problem.
* System should have more than 1 functions deployed on the same src-bkt * KV rebalance should be triggered * As part of this KV rebalance, when eventing rebalance begins, undeploy one of the functions on src-bkt * If system hits this race condition, rebalance will be stuck and will fail after 10 mins. customer deployment ran into this issue twice. |
| Link | This issue blocks MB-40528 [ MB-40528 ] |
| Labels | approved-for-6.6.1 |
Thank you Wayne Siu!
We shall target have the fix ready before RC2 is picked.
Build couchbase-server-6.6.1-9205 contains eventing commit 2c94b27 with commit message:
MB-42961: Unwatch bucket when all handlers listening on the bucket gets undeployed/paused
Build couchbase-server-6.6.1-9206 contains eventing commit 5532664 with commit message:
MB-42961: Terminate callbacks if consumer is terminating and unwatch src bkt if we can not watch meta bkt
Build couchbase-server-6.6.1-9207 contains eventing commit 4148765 with commit message:
MB-42961: Refresh bucket object when its out of sync with the bucket uuid
| Resolution | Fixed [ 1 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Status | Resolved [ 5 ] | Closed [ 6 ] |
| Link | This issue relates to CBSE-9296 [ CBSE-9296 ] |
Ritam Sharma Vikas Chaudhary We should fix this in 6.6.1. Request inclusion, Thank you!
Fix is to ref count the src-bkt references accurately and close a reference only after all functions against it are undeployed/paused. We will propose a fix in a day or two.