RSS growth in pillowfight leading to Linux oom killer

Description

During an overnight load test I observed that pillowfight was terminated by Linux out-of-memory killer (see attachment).

At the time of the OOM-killer being invoked, pillowfight's VSZ had grown to 7603584 pages or 29,701 MB, and RSS to 28,446 MB:

Pillowfight command:

I re-ran pillowfight with jemalloc as the memory allocator (via LD_PRELOAD); which suggests there might be some leaks of tracing span-related objects:

Environment

Version=3.2.4_1_gf6822fcc85 AWS Graviton2 AArch64, Amazon EC2 c6gd.4xlarge

Gerrit Reviews

None

Release Notes Description

None

Attachments

6

Activity

Show:

CB robot December 19, 2023 at 1:39 AM

Build couchbase-server-7.1.6-3969 contains libcouchbase commit 75dd370 with commit message:
: pillowfight: deallocate all memory during shutdown

CB robot April 12, 2023 at 7:11 AM

Build couchbase-server-7.2.0-5304 contains libcouchbase commit 75dd370 with commit message:
: pillowfight: deallocate all memory during shutdown

CB robot April 10, 2023 at 11:04 PM

Build couchbase-server-8.0.0-1291 contains libcouchbase commit 75dd370 with commit message:
: pillowfight: deallocate all memory during shutdown

CB robot April 10, 2023 at 7:44 AM

Build couchbase-server-7.5.0-4098 contains libcouchbase commit 75dd370 with commit message:
: pillowfight: deallocate all memory during shutdown

Dave Rigby February 7, 2023 at 3:15 PM

Note this is still a problem with 3.2.5; such that I cannot run pillowfight for more than ~5minutes when driving ~100k operations per second on a 32GB machine...

Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Story Points

Components

Reporter

Affects versions

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created December 14, 2021 at 3:01 PM
Updated December 19, 2023 at 1:39 AM
Instabug