[CBAS DP2] CBAS does not cleanup and restart ingestion in event of a KV rollback

Build : 647

In case of a KV rollback, the expected behavior for Analytics is to clean up and restart ingestion from scratch. Currently it does not do it, hence there is a mismatch in the no. of items between the KV bucket and CBAS bucket.

Testrunner testcase :
./testrunner -i cbas.ini -t cbas.cbas_bucket_operations.CBASBucketOperations.test_ingestion_after_kv_rollback,cb_bucket_name=default,cbas_bucket_name=default_bucket,cbas_dataset_name=default_ds,items=10000,nodes_init=2,cluster=D,D

Steps :
1. Setup a 2 node KV cluster- nodeA and nodeB
2. Create a bucket and load some docs on KV
3. Create a CBAS bucket, a shadow dataset and connect it to the KV bucket
4. Ensure that the # items matches in the KV bucket and CBAS bucket
5. Stop Persistence on Node A & Node B on KV
6. Perform mutations on KV side. Since persistence is disabled, the mutations will remain in memory. Preferably do deletes or creates.
7. Again get the count of items in KV bucket and CBAS bucket. It should still match.
8. Kill memcached on NodeA, so that nodeB becomes the master
9. Enable persistence on NodeB
10. Failover NodeB.
11. By the time nodeA comes up, nodeB should be down, and before going down, it would push some mutations that were done in Step 6 to persistence. It might not be able to push everything down to persistence. NodeA would fetch the mutations from persistence, and would inform the DCP clients about the rollback.
12. CBAS should cleanup the dataset and restart ingestion in case of rollback.

This doesnt happen, and there is a mismatch in no. of items between KV and CBAS. Logs attached.