200% beam.smp usage during rebalance after failover ( cluster becomes unresponsive) - amazon ec2

this happened while running failover tests :

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1184 membase 20 0 1192m 120m 2472 S 195.0 1.6 16:48.01 beam.smp
1224 membase 20 0 2878m 2.0g 3048 S 3.9 28.1 10:49.18 memcached
390 root 20 0 0 0 0 S 2.0 0.0 0:08.14 jbd2/xvda1-8
1 root 20 0 19108 1388 1144 S 0.0 0.0 0:00.35 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
7 root 20 0 0 0 0 S 0.0 0.0 0:01.36 ksoftirqd/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
9 root 20 0 0 0 0 S 0.0 0.0 0:00.78 events/0
10 root 20 0 0 0 0 S 0.0 0.0 0:00.44 events/1
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset
12 root 20 0 0 0 0 S 0.0 0.0 0:00.01 khelper
76 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
77 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr

1- build 30 node cluster with
2- failover 3 nodes , 30->27
3- rebalance

loop through 1-3

this happened when rebalancing from i think 18->15

attached the diags from mbbrowselog and also the erlang crash dump file generated by running killall -SIGUSR on beam.smp