Analytics Longevity : Not all items ingested by CBAS

Analytics Build : 5.0.0-639

We have a new longevity test that does the following:

1. 4 node cluster = 2 kv, 2 analytics
2. Create 4 CB buckets
3. Create 4 corresponding CBAS buckets
4. Load some docs in all buckets for sometime
5. Create 4 corresponding CBAS buckets, datasets and connect to the CB buckets
6. Run KV ops (with creates, deletes, updates) continuously on all buckets
7. Run analytic queries that touches both the analytics nodes and covers all datasets continuously. I will deliberately pump in queries with a small batch size so as to not run into query rejections due to full queue.
8. Stop after a specified time.

This test was run for 10 hrs. At the end of it, I ran a simple count query for all the analytics datasets, and found that the number of items in analytics do not match those in the KV buckets. The datasets are non-filtered, i.e. they should ingest everything.

CB bucket & num items CBAS bucket & num items
default : 3505276 ds1 - 2195054
other-1 : 3289230 ds2 - 2183257
other-2 : 5225217 ds3 - 2591241
other-3: 5152329 ds4 - 3328123

Also, there are tons of the following warnings in the analytics node. Maybe that is related.

2017-08-09T18:30:49.150-07:00 WARN CBAS.server.QueryServiceServlet [HttpExecutor(port:8095)-14] Error flushing output writer

Attached analytics logs from both the nodes. cbcollectinfo location mentioned below.

The env is live if you want to take a look :
http://172.23.108.162:8091/ - KV node1
http://172.23.108.17:8091/ - CBAS node1
http://172.23.108.46:8091/ - CBAS node2