Details
Description
WE have a cluster of about 100 apache+php servers accessing a couchbase cluster.
There are between 30K-100K operations to the cluster depending on time of the day,
of which 0.5K-1.5K are "set" operations and rest are "get"s.
Every several minutes some random php server starts to get bursts of failed "set" opertations, which return "false" (while the rest of the servers are just fine).
These burst may lasts from a several seconds to several minutes (also same node has high chances of getting several such burst every several minutes), during these burst only some portion of the "set"s fail.
Then inspecting the getResultCode it always returns 0 in such cases, and no CouchbaseException is thrown.
There are other cases then a few operations timeout, but in this case and exception is thrown and we can retrieve 23 result code. So the issue above is really weird.
relevant php couchbase ext 1.2.1 configs:
couchbase.config_cache = "/ephemeral/php/couchbase"
couchbase.skip_config_errors_on_connect = On
Any clues for debugging the issue?