Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Magma flooding log in case of state file deletion failure

Description

Supportal faced downtime caused by their three-node 7.1.2 couchbase cluster. The Magma bucket was flooding the logs and constantly retrying a file delete on one of their kv nodes as reported possibly causing the slowness and preventing debugging of the issue.

2024-01-18T10:59:30.802799+00:00 WARNING [(default) magma_5/kvstore-773/rev-000000002/localIndex]Failed to delete state file 5972 reason:IOError: /mnt/couchbase/data/default/magma.5/kvstore-773/rev-000000002/localIndex/state.5972: Unable to delete : File not found err=No such file or directory. Closest non-empty parent directory: /mnt/couchbase/data/default/magma.5/kvstore-773/rev-000000002/localIndex offset:0 len:0

This issue was probably caused by an intermittent filesystem issue resulting in a file remove/unlink returning a bad status despite the file being deleted causing Magma to get stuck in a retry loop trying to delete the file. Rebooting the node fixed the issue. A defensive fix is required to prevent Magma from flooding the logs and potentially pinning a CPU core causing slowness.

 

Components

Affects versions

Fix versions

Labels

Environment

None

Link to Log File, atop/blg, CBCollectInfo, Core dump

None

Release Notes Description

None

Activity

CB robot January 23, 2024 at 3:19 PM

Build couchbase-server-8.0.0-1536 contains magma commit de6ff73 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-60444#icft=MB-60444 magma: Check if state file exists before attempting deletion

CB robot January 23, 2024 at 12:58 AM

Build capella-analytics-1.0.0-1216 contains magma commit de6ff73 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-60444#icft=MB-60444 magma: Check if state file exists before attempting deletion

CB robot January 22, 2024 at 6:41 PM

Build couchbase-server-7.6.0-2040 contains magma commit de6ff73 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-60444#icft=MB-60444 magma: Check if state file exists before attempting deletion

CB robot January 22, 2024 at 6:17 PM

Build couchbase-server-7.6.1-3048 contains magma commit de6ff73 with commit message:
https://couchbasecloud.atlassian.net/browse/MB-60444#icft=MB-60444 magma: Check if state file exists before attempting deletion

Apaar Gupta January 22, 2024 at 4:48 PM

Merged fix

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

Unknown

Triage

Untriaged

Due date

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created January 18, 2024 at 4:44 PM
Updated September 12, 2024 at 6:12 PM
Resolved January 22, 2024 at 4:48 PM
Instabug