plasma: Storage log grows indefinitely

The plasma log grows indefinitely around 16MB in size and it grows until disk fills up. It appears to be a rare occurrence.

When the issue happens, the lss_gc_blk_bytes_read is much lower than the bytes written. Which indicates that the growth of space was caused by single log cleaner run. If there were multiple runs, lss_gc_blk_bytes_read should be comparable to bytes written. The entire index data size is only 16MB. 

It is likely that relocation part of cleaner ran into a loop which never came out causing disk to fill up.

bytes_incoming": 195780,
"bytes_written": 3386680905728,
"write_amp": 1422.24,
"write_amp_avg": 17298400.79,
"lss_fragmentation": 99,
"lss_data_size": 7554799,
"lss_used_space": 3386664644608,
"lss_num_reads": 1826,
"lss_read_bs": 51385149,
"lss_blk_read_bs": 58168826,
"lss_gc_num_reads": 1826,
"lss_gc_reads_bs": 34322320,
"lss_blk_gc_reads_bs": 41103360,
"lss_rdr_reads_bs": 0,
"lss_blk_rdr_reads_bs": 0,