Big performance drop when doing small proportion of bgfetches

• get mc-loader from git://github.com/alk/mc-loader.git and built it (note you'll need some sasl-aware version of libmemcached. Maybe our fork, maybe stock will work. You'll need to edit makefile manually, sorry)

• run this:

./generate-keyset.rb -k 3000000 -a '

{"a":"asddddddddddddddddddddddddddddddddddddddddddddddddasdddddddddsdfkjs;dlfkjas;dlkfj;asdlkjfa;sdlkf;dj;kj;ksjkdfjg;kjdg;lkdjfg;lkg;ksf;jflj;gldlk;lfjgskj;lksg;lfg;ksf;ldsflkjlgsdkfj;kfjslfjsljskdjlfjsljljlskjksgskdgj"}

' | \time ./parallelize-lines 12 ./mc-loader lh:12001 -

It means 3e6 k/v-s will be generated and piped into paralellize-lines. Parallize-lines will spawn 12 mc-loaders and split stream of lines into them evenly (round-robin). mc-loader as you can see is asked to send stuff to moxi and read k/v-s from stdin.

With this setup on my box I can easily hit 65k ops/sec. When some values are non-resident but still are in page cache (my box has 32 gigs of ram) I'm seeing get rate (add "check" to mc-loader args for GETs) of varying rapidly and is much smaller (we're fetching items from page cache, that's ram disk essentially): http://i.imgur.com/p53CZ.png

NOTE: I've seen much lower rate. Perhaps that was multiple nodes, in screenshot above it's just one.