Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
6.0.1
-
component cluster
-
Untriaged
-
-
Unknown
Description
Build : 6.0.1-1992
Test : -test tests/2i/test_idx_rebalance_replica_vulcan_kv_opt.yml -scope tests/2i/scope_idx_rebalance_replica_vulcan.yml
Scale: 3
Iteration : 3
Seeing memcached getting OOM killed 2 times on 2 different nodes - 172.23.104.16 & 172.23.104.18
On .18,
[Fri Dec 7 11:29:32 2018] Out of memory: Kill process 26360 (memcached) score 689 or sacrifice child
[Fri Dec 7 11:29:32 2018] Killed process 26360 (memcached) total-vm:20996316kB, anon-rss:16878860kB, file-rss:0kB, shmem-rss:0kB
On .16
[Fri Dec 7 12:05:41 2018] Out of memory: Kill process 30725 (memcached) score 684 or sacrifice child
[Fri Dec 7 12:05:41 2018] Killed process 30725 (memcached) total-vm:20502756kB, anon-rss:16746964kB, file-rss:0kB, shmem-rss:0kB
[Fri Dec 7 12:05:41 2018] memcached: page allocation failure: order:0, mode:0x2015a
[Fri Dec 7 12:05:41 2018] CPU: 1 PID: 30725 Comm: memcached Not tainted 3.10.0-693.5.2.el7.x86_64 #1
On 16, projector also got OOM killed around the same time. Tracking this issue via a different bug.
[Fri Dec 7 12:05:41 2018] Out of memory: Kill process 7008 (projector) score 177 or sacrifice child
[Fri Dec 7 12:05:41 2018] Killed process 7008 (projector) total-vm:4841764kB, anon-rss:4339896kB, file-rss:0kB, shmem-rss:0kB
The test was rebalancing out a KV node when the OOM killing on .18 happened. It was rebalancing in a KV node when the OOM killing on .16 happened.