[CX] NCServices running even after stopping cluster using stop-sample-cluster.sh

1. Tried starting cluster using :

./samples/local/bin/start-sample-cluster.sh 

CLUSTERDIR=/root/cbas-install/cbas/samples/local

INSTALLDIR=/root/cbas-install/cbas

LOGSDIR=/root/cbas-install/cbas/samples/local/logs

ERROR: sample cluster address (localhost:8095) already in use

2. Checked netstat and jps:

root@ubuntu:~/cbas-install/cbas# netstat -anp | grep -i 8095

tcp        0      0 0.0.0.0:8095            0.0.0.0:*               LISTEN      20672/java      

tcp        0      0 127.0.0.1:56327         127.0.0.1:8095          TIME_WAIT   -               

unix  3      [ ]         STREAM     CONNECTED     7538095  6496/moxi           

root@ubuntu:~/cbas-install/cbas# jps

20672 CCDriver

6978 Jps

20803 NCDriver

20797 NCDriver

20670 NCService

20671 NCService

3. Tried shutting down cluster and then checked jps:

root@ubuntu:~/cbas-install/cbas# curl -v -data http://localhost:8095/analytics/shutdown

* Hostname was NOT found in DNS cache

*   Trying 127.0.0.1...

* Connected to localhost (127.0.0.1) port 8095 (#0)

> POST /analytics/shutdown HTTP/1.1

> User-Agent: curl/7.35.0

> Host: localhost:8095

> Accept: */*

> Content-Length: 3

> Content-Type: application/x-www-form-urlencoded

> 

* upload completely sent off: 3 out of 3 bytes

< HTTP/1.1 202 Accepted

< Date: Fri, 21 Oct 2016 05:45:32 GMT

< Content-Type: application/json;charset=utf-8

< Content-Length: 799

* Server Jetty(9.3.11.v20160721) is not blacklisted

< Server: Jetty(9.3.11.v20160721)

< 

{

    "cluster": {

        "metadata_node": "red",

        "ncs": [

            {

                "ncservice_pid": 20671,

                "node_id": "red",

                "partitions": [{

                    "active": true,

                    "partition_id": "partition_0"

                }],

                "pid": 20803,

                "state": "ACTIVE"

            },

            {

                "ncservice_pid": 20670,

                "node_id": "blue",

                "partitions": [{

                    "active": true,

                    "partition_id": "partition_1"

                }],

                "pid": 20797,

                "state": "ACTIVE"

            }

        ],

        "state": "ACTIVE"

    },

    "date": "Thu Oct 20 22:45:32 PDT 2016",

    "status": "SHUTTING_DOWN"

* Connection #0 to host localhost left intact

}root@ubuntu:~/cbas-install/cbas# jps

7073 Jps

20670 NCService

20671 NCService

4. Tried stopping cluster and then checked jps:

root@ubuntu:~/cbas-install/cbas# ./samples/local/bin/stop-sample-cluster.sh 

WARNING: sample cluster does not appear to be running, will attempt to wait for

         CCDriver to terminate if running.

root@ubuntu:~/cbas-install/cbas# jps

7166 Jps

20670 NCService

20671 NCService

5. All services should have been stopped but I still see these 2 NCServices running. I have taken jstack of these two, attaching them to the bug.Also attaching cbas logs.

6. Error seen in blue-service.log:

2016-10-20T22:45:32.398-07:00 ERRO CBAS.dataset.ResultStateSweeper [Executor-5:blue] Result cleaner thread interrupted, shutting down.

org.apache.hyracks.api.exceptions.HyracksDataException: Task java.util.concurrent.FutureTask@2f949883 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 7, active threads = 6, queued tasks = 0, completed tasks = 4569]

        at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:228)

        at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.deinitialize(SuperActivityOperatorNodePushable.java:165)

        at org.apache.hyracks.control.nc.Task.run(Task.java:315)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2f949883 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 7, active threads = 6, queued tasks = 0, completed tasks = 4569]

        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)

        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)

        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)

        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)

        at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:215)

        ... 5 more

org.apache.hyracks.api.exceptions.HyracksDataException: Task java.util.concurrent.FutureTask@6cb84ed5 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 6, active threads = 6, queued tasks = 0, completed tasks = 4569]

        at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:228)

        at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.deinitialize(SuperActivityOperatorNodePushable.java:165)

        at org.apache.hyracks.control.nc.Task.run(Task.java:315)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6cb84ed5 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 6, active threads = 6, queued tasks = 0, completed tasks = 4569]

        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)

        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)

        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)