Race condition between durability settings and pessimistic locks

Problem:

Sometimes an insert fails with KeyExistsError when a just inserted record is being locked in the other thread and durability settings are set to either persist or replicate a record on insert.

Background:

We have a distributed system where new records are massively being inserted.  In this system we use N1QL queries which regularly fetch a subset of records from a GSI index in order to update them.  We use pessimistic locks (bucket.lock function) in order to synchronize different workers so that only one of them have access to a record at a time.  Due to system requirements optimistic locks though CAS was not an option.

In the above scenario we regularly observe a situation when a record insert fails with KeyExistsError although in fact that record didn't exist before.  The worst thing is that a record is in fact inserted (and locked), so that is a false message.

Steps to Reproduce:

1. Install a travel-sample bucket (or create a password-less empty bucket with that name).
2. Run an attached script.
3. It will print a lot of 'HERE IS A BUG' messages

Analysis:

With a current CouchBase implementation it's possible to lock a record while it's being inserted if either persist_to or replicate_to durability settings are non-zero (the best is to set both of them).  An insert doesn't account for such use case and fails with KeyExistsError instead of returning a success.