fdb iterator skips documents that should be returned, while the writer adds new documents

This issue was observed in Couchbase Lite unit test. The following is more details from Jens:

-------------------------------------------------------------------------------------------------------------------------

I’ve run into a nasty problem where an fdb_iterator skips over a document that should be returned. The behavior is intermittent, but I have a test that can reproduce it reliably, though it takes a different number of iterations every time.

The test has two threads that operate on one database. The first keeps adding documents to the database. All documents are consecutively numbered “doc-%05d”, where the number keeps increasing. It adds some number of documents (usually 1 to 10) in a transaction, pauses, and repeats.

A second thread repeatedly (1) creates an iterator starting at the first key, (2) advances the iterator until it reaches the end, reading every document and validating its key, then (3) frees the iterator.

What happens after some iterations of this is that the second thread fails an assertion because the document keys aren’t consecutive — for example it gets “doc-123” followed by “doc-127”. The missing document(s) are in the database, they’re just being skipped.

These threads use the same fdb_file_handle, but use a mutex to avoid using the handle at the same time. Thread 1 holds the mutex the whole time it has the transaction, and thread 2 holds the mutex while it creates the iterator. I’m not getting any HANDLE_BUSY errors, and I’ve reviewed the locking code and I’m pretty confident in it. So I think the same bug could be reproduced with a single thread just by interleaving the same calls made by the two threads.

This is the behavior with the latest stable commit (d16733d9). (I was previously using a commit from the stable branch as of a few weeks ago, and getting a different problem where the first thread’s transaction was sometimes losing some of the changes.)