[System Test] Analytics rebalance hangs after node remove

System test involves the following steps in loop for 24 hours

Steps:

• Create 4 node cluster with 2 KV,2 CBAS nodes
• Create 4 data buckets and 4 corresponding buckets in CBAS
• Create 4 data sets and index on each bucket
• Start populating data 
• Run SELECT query (Note : These queries are running at the rate of 12 queries/sec during whole test  on each bucket)
• Connect all buckets and wait for some time for ingestion 
• Disconnect all the bucket :- Fails as node is busy
• Remove one node and rebalance <- Rebalance stuck here

 
NOTE: we are constantly populating data and running query during test also we are not changing node where we are  queries.

Jenkins job is still running http://qa.sc.couchbase.com/job/component_systest_launcher/55/console

Cluster is available for debugging and rebalance is also hanged since 1 hr