Magma flooding log in case of state file deletion failure

Supportal faced downtime caused by their three-node 7.1.2 couchbase cluster. The Magma bucket was flooding the logs and constantly retrying a file delete on one of their kv nodes as reported here possibly causing the slowness and preventing debugging of the issue.

2024-01-18T10:59:30.802799+00:00 WARNING [(default) magma_5/kvstore-773/rev-000000002/localIndex]Failed to delete state file 5972 reason:IOError: /mnt/couchbase/data/default/magma.5/kvstore-773/rev-000000002/localIndex/state.5972: Unable to delete : File not found err=No such file or directory. Closest non-empty parent directory: /mnt/couchbase/data/default/magma.5/kvstore-773/rev-000000002/localIndex offset:0 len:0

This issue was probably caused by an intermittent filesystem issue resulting in a file remove/unlink returning a bad status despite the file being deleted causing Magma to get stuck in a retry loop trying to delete the file. Rebooting the node fixed the issue. A defensive fix is required to prevent Magma from flooding the logs and potentially pinning a CPU core causing slowness.