Slow tap responses from couchbase db observed on sync-gateway

The output from sync-gateway's expvar (http://172.23.96.63:4985/_expvar) shows that a large number of tap responses from Couchbase db takes over 1 minute. For example:
"lag-total-28900ms": 8676, (8676 tap responses took 28 seconds)
"lag-total-56400ms": 21,
"lag-total-20300ms": 17632,
"lag-total-26900ms": 6564,
"lag-queue-49100ms": 45,

This happen when two sync-gateways, each handles 10K users, are running in parallel. This happen about one hour after the test started.

Attached are the two couchbase console screen shots, and the expvar from sync_gateway that shows the above tap responses.

The cbcollect_info zip file is too large to attach to this ticket. It can be accessed from http://172.23.106.228/20140424-164336-8cpu8g-3gw3gl3db-u10000-c16-fail/cblog-64.zip