Description
What is the problem?
Quite often we see customers hit issues where the archive/staging directory has a stale lockfile in it. Usually they get cleaned up due to the fact that they're from the same hostname and the PID no longer exists, but these conditions do not always hold - e.g. if NFS is involved.
We should aim to clean the lockfiles up on exit whenever possible
What is the solution?
We should make lockfile unlocks not error if the lockfile no longer exists and then, whenever we lock an archive/staging dir we should defer an unlock. That means if we error or we panic the lockfile should be cleaned up.