Details
-
Bug
-
Resolution: Fixed
-
Major
-
CBAS DP
-
None
-
Untriaged
-
Unknown
-
CX Sprint 29, CX Sprint 30
Description
In order to reproduce MB-21435,I added Tills suggestions in script and hit this issue.
1. Tried starting cluster using :
|
./samples/local/bin/start-sample-cluster.sh
|
CLUSTERDIR=/root/cbas-install/cbas/samples/local
|
INSTALLDIR=/root/cbas-install/cbas
|
LOGSDIR=/root/cbas-install/cbas/samples/local/logs
|
|
ERROR: sample cluster address (localhost:8095) already in use
|
|
2. Checked netstat and jps:
|
root@ubuntu:~/cbas-install/cbas# netstat -anp | grep -i 8095
|
tcp 0 0 0.0.0.0:8095 0.0.0.0:* LISTEN 20672/java
|
tcp 0 0 127.0.0.1:56327 127.0.0.1:8095 TIME_WAIT -
|
unix 3 [ ] STREAM CONNECTED 7538095 6496/moxi
|
root@ubuntu:~/cbas-install/cbas# jps
|
20672 CCDriver
|
6978 Jps
|
20803 NCDriver
|
20797 NCDriver
|
20670 NCService
|
20671 NCService
|
|
3. Tried shutting down cluster and then checked jps:
|
root@ubuntu:~/cbas-install/cbas# curl -v -data http://localhost:8095/analytics/shutdown
|
* Hostname was NOT found in DNS cache
|
* Trying 127.0.0.1...
|
* Connected to localhost (127.0.0.1) port 8095 (#0)
|
> POST /analytics/shutdown HTTP/1.1
|
> User-Agent: curl/7.35.0
|
> Host: localhost:8095
|
> Accept: */*
|
> Content-Length: 3
|
> Content-Type: application/x-www-form-urlencoded
|
>
|
* upload completely sent off: 3 out of 3 bytes
|
< HTTP/1.1 202 Accepted
|
< Date: Fri, 21 Oct 2016 05:45:32 GMT
|
< Content-Type: application/json;charset=utf-8
|
< Content-Length: 799
|
* Server Jetty(9.3.11.v20160721) is not blacklisted
|
< Server: Jetty(9.3.11.v20160721)
|
<
|
{
|
"cluster": {
|
"metadata_node": "red",
|
"ncs": [
|
{
|
"ncservice_pid": 20671,
|
"node_id": "red",
|
"partitions": [{
|
"active": true,
|
"partition_id": "partition_0"
|
}],
|
"pid": 20803,
|
"state": "ACTIVE"
|
},
|
{
|
"ncservice_pid": 20670,
|
"node_id": "blue",
|
"partitions": [{
|
"active": true,
|
"partition_id": "partition_1"
|
}],
|
"pid": 20797,
|
"state": "ACTIVE"
|
}
|
],
|
"state": "ACTIVE"
|
},
|
"date": "Thu Oct 20 22:45:32 PDT 2016",
|
"status": "SHUTTING_DOWN"
|
* Connection #0 to host localhost left intact
|
}root@ubuntu:~/cbas-install/cbas# jps
|
7073 Jps
|
20670 NCService
|
20671 NCService
|
|
4. Tried stopping cluster and then checked jps:
|
root@ubuntu:~/cbas-install/cbas# ./samples/local/bin/stop-sample-cluster.sh
|
WARNING: sample cluster does not appear to be running, will attempt to wait for
|
CCDriver to terminate if running.
|
root@ubuntu:~/cbas-install/cbas# jps
|
7166 Jps
|
20670 NCService
|
20671 NCService
|
|
|
5. All services should have been stopped but I still see these 2 NCServices running. I have taken jstack of these two, attaching them to the bug.Also attaching cbas logs.
|
|
6. Error seen in blue-service.log:
|
2016-10-20T22:45:32.398-07:00 ERRO CBAS.dataset.ResultStateSweeper [Executor-5:blue] Result cleaner thread interrupted, shutting down.
|
org.apache.hyracks.api.exceptions.HyracksDataException: Task java.util.concurrent.FutureTask@2f949883 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 7, active threads = 6, queued tasks = 0, completed tasks = 4569]
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:228)
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.deinitialize(SuperActivityOperatorNodePushable.java:165)
|
at org.apache.hyracks.control.nc.Task.run(Task.java:315)
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
|
at java.lang.Thread.run(Thread.java:745)
|
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2f949883 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 7, active threads = 6, queued tasks = 0, completed tasks = 4569]
|
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
|
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
|
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
|
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:215)
|
... 5 more
|
org.apache.hyracks.api.exceptions.HyracksDataException: Task java.util.concurrent.FutureTask@6cb84ed5 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 6, active threads = 6, queued tasks = 0, completed tasks = 4569]
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:228)
|
at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.deinitialize(SuperActivityOperatorNodePushable.java:165)
|
at org.apache.hyracks.control.nc.Task.run(Task.java:315)
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
|
at java.lang.Thread.run(Thread.java:745)
|
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6cb84ed5 rejected from java.util.concurrent.ThreadPoolExecutor@426e1f4f[Shutting down, pool size = 6, active threads = 6, queued tasks = 0, completed tasks = 4569]
|
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
|
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
|
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
|
|
|