Data loss is observed due to wal corruption.

Scenario:
1. 2 node cluster, 1 default bucket running on magma with replica=1
2. load data(100k)
3. check ep_queue_size=0, this ensures everything is persisted. Verify all stats like active item count, replica item count
4. kill memcached on 1 node(.220)
5. load another 100k
6. repeat step 2, Step 2 fails on 16th iteration.

memcached logs on .220 shows:
CRITICAL WALIterator corruption error:Corruption: 8:48586752

QE test:

-t magma.magma_crash_tests.MagmaCrashTests.test_crash_magma_n_times,num_items=100000,infra_log_level=debug,skip_cleanup=true,rerun=False,doc_size=1024,nodes_init=2,num_crashes=50,sdk_timeout=60,bucket_storage=magma