[CBAS] Analytics processes hang and clients get socket timed out

Build : 597

Issue : In a series of testcases for disconnecting bucket with different permutations, at a certain point, we start seeing
ERROR - socket error while connecting to http://10.111.151.102:8095/analytics/service error timed out

This is 100% reproducible with testrunner. Follow the steps given below to reproduce the issue.
1. Create a conf file like

cbas.cbas_functional_tests.CBASFunctionalTests: 

    # Disconnect Bucket tests

    test_disconnect_bucket,cluster=D

    test_disconnect_bucket,disconnect_if_connected=True,cluster=D

    test_disconnect_bucket,cbas_bucket_name_invalid=travel_invalid,error=A bucket with this name travel_invalid doesn't exist.,cluster=D

  test_disconnect_bucket,cbas_bucket_name_invalid=travel_invalid,disconnect_if_connected=True,error=A bucket with this name travel_invalid doesn't exist.,cluster=D

    test_disconnect_bucket_already_disconnected,error=The bucket travel is not connected.,cluster=D

    test_disconnect_bucket_already_disconnected,disconnect_if_connected=True,cluster=D

2. Create an ini file like

[global]

username:root

password:couchbase

#username:Administrator

#password:Membase123

port:8091

n1ql_port:8093

index_port:9102

data_path=/data

index_path=/data

[membase]

rest_username:Administrator

rest_password:password

[servers]

1:_1

[_1]

ip:10.111.151.101

services:kv,n1ql,index

[cbas]

ip:10.111.151.102

port:8095

username:root

password:couchbase

3. Setup testrunner on your Mac - Follow the instructions here : https://github.com/couchbase/testrunner

4. Clone testrunner - https://github.com/couchbase/testrunner

5. Run testrunner - cd testrunner; ./testrunner.py -i <your ini file> -c <your conf file>

There should be failures from the 5th test.

Analytics logs are attached.

At the point of failure, the following can be seen in the red-service.log and blue-service.log
2017-03-16T11:08:43.522Z INFO CBAS.runtime.BucketOperatorNodePushable [Executor-30:red] (Default.travel(CouchbaseMetadataExtension))[0]:BucketOperatorDescriptor aborting, acquiring start semaphore...
2017-03-16T11:10:48.107Z INFO CBAS.work.WorkQueue [Worker:red] Executing: ApplicationMessage: nodeID: red
2017-03-16T11:10:48.107Z INFO CBAS.messaging.NCMessageBroker [Worker:red] Received message: ActiveManagerMessage
2017-03-16T11:10:48.109Z INFO CBAS.runtime.BucketOperatorNodePushable [Executor-31:red] (Default.travel(CouchbaseMetadataExtension))[0]:BucketOperatorDescriptor aborting, acquiring start semaphore...
2017-03-16T11:12:05.379Z INFO CBAS.work.WorkQueue [Worker:red] Executing: ApplicationMessage: nodeID: red
2017-03-16T11:12:05.380Z INFO CBAS.messaging.NCMessageBroker [Worker:red] Received message: ActiveManagerMessage
2017-03-16T11:12:05.382Z INFO CBAS.runtime.BucketOperatorNodePushable [Executor-32:red] (Default.travel(CouchbaseMetadataExtension))[0]:BucketOperatorDescriptor aborting, acquiring start semaphore...