Details
-
Improvement
-
Resolution: Fixed
-
Major
-
None
-
None
-
None
-
1
Description
Bucket deletion happens synchronously in ns_orchestrator in the idle state and therefore any attempt to failover will just be queued in ns_orchestrator gen_statem message queue and will not make progress.
Opening this improvement based on the discussion below with Dave.
Hareen: Had a simple question: It looks today when we delete a bucket failover can be stuck from progressing since we delete the bucket synchronously in the idle state in ns_orchestrator; I was just wondering if it was by design since I was going to delete the bucket at the tail of a successfully completed pause bucket.
|
|
|
Dave: I think it’s probably an oversight - though perhaps it’s related to bucket deletion being “best effort” today anyway (and so in this case might be “by design” because we don’t expect buckets to be deleted frequently.) But in any case, bucket deletion can take multiple minutes and failover should probably interrupt it. Ideally bucket deletion would be a long-lived transaction i.e. we’d mark the bucket as deleting and then only remove it when all the nodes have finished clean up. This would allow us to cleanly interrupt bucket deletion and run a failover.
|
All in all: this is an issue and a potentially significant issue in serverless. We should create an improvement to allow failover to interrupt bucket deletion.
|
11:50
|
Can you create the improvement?
|