We're leaving some disk performance on the table

I'm pretty sure this wasn't case few month ago.

There's around 2x difference in write throughput with barriers on and off.

That indicates we're doing fsync calls a bit too often.

My thinking is with right size of transactions we should be able to fsync seldom enough so that effect of fsync (actually waiting until stuff hits platter) is negligible.

I should also note that this difference can be due to FS metadata ops which we're doing quite a bit as well (all those file size modification). If this difference is due to underlying FS metadata transactions then there's nothing we can do in short term.