Details
-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
3.2.4, 3.2.5
-
None
-
Version=3.2.4_1_gf6822fcc85
AWS Graviton2 AArch64, Amazon EC2 c6gd.4xlarge
-
1
Description
During an overnight load test I observed that pillowfight was terminated by Linux out-of-memory killer (see attachment).
At the time of the OOM-killer being invoked, pillowfight's VSZ had grown to 7603584 pages or 29,701 MB, and RSS to 28,446 MB:
Dec 13 17:56:16 ip-172-31-37-130 kernel: Tasks state (memory values in pages):
|
Dec 13 17:56:16 ip-172-31-37-130 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
|
Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 8907] 1000 8907 7603584 7282351 58626048 0 0 cbc-pillowfight
|
...
|
Dec 13 17:56:16 ip-172-31-37-130 kernel: [ 20142] 1000 20142 1776314 576103 6610944 0 0 memcached
|
Pillowfight command:
cbc-pillowfight -u Administrator -P asdasd -U couchbase://127.0.0.1:12000/default -m 1024 -M 1024 --json --random-body -I 500000000 --num-threads 32
|
I re-ran pillowfight with jemalloc as the memory allocator (via LD_PRELOAD); which suggests there might be some leaks of tracing span-related objects: pillowfight.prof.svg
pillowfight.objects-prof.svg
Attachments
Gerrit Reviews
| For Gerrit Dashboard: CCBC-1527 | ||||||
|---|---|---|---|---|---|---|
| # | Subject | Branch | Project | Status | CR | V |
| 174584,4 | CCBC-1527: pillowfight: deallocate all memory during shutdown | master | libcouchbase | Status: MERGED | +2 | +1 |