[INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building cbas-cbserver-test 1.0.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ cbas-cbserver-test --- [INFO] [INFO] --- formatter-maven-plugin:2.0.1:format (default) @ cbas-cbserver-test --- [INFO] Using 'UTF-8' encoding to format source files. [INFO] Number of files to be formatted: 20 [INFO] Successfully formatted: 0 file(s) [INFO] Fail to format: 0 file(s) [INFO] Skipped: 20 file(s) [INFO] Read only skipped: 0 file(s) [INFO] Approximate time taken: 1s [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ cbas-cbserver-test --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ cbas-cbserver-test --- [INFO] [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ cbas-cbserver-test --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/cbas-cbserver-test/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ cbas-cbserver-test --- [INFO] No sources to compile [INFO] [INFO] --- maven-checkstyle-plugin:2.17:check (verify-style) @ cbas-cbserver-test --- [INFO] [INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ cbas-cbserver-test --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 687 resources [INFO] [INFO] --- maven-dependency-plugin:3.1.1:unpack-dependencies (unpack-cbas-ingestion-test-tests) @ cbas-cbserver-test --- [INFO] com.couchbase.analytics:cbas-ingestion-test:test-jar:tests:1.0.0-SNAPSHOT already exists in destination. [INFO] [INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ cbas-cbserver-test --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-dependency-plugin:3.1.1:analyze-only (default) @ cbas-cbserver-test --- [INFO] No dependency problems found [INFO] [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ cbas-cbserver-test --- [JENKINS] Recording test results [WARNING] Attempt to (de-)serialize anonymous class hudson.maven.reporters.BuildInfoRecorder$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ [INFO] [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ cbas-cbserver-test --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] [INFO] --- maven-site-plugin:3.7.1:attach-descriptor (attach-descriptor) @ cbas-cbserver-test --- [INFO] Skipping because packaging 'jar' is not pom. [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ cbas-cbserver-test --- [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit4/2.21.0/surefire-junit4-2.21.0.pom [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit4/2.21.0/surefire-junit4-2.21.0.pom (4 KB at 14.0 KB/sec) [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.21.0/surefire-providers-2.21.0.pom [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-providers/2.21.0/surefire-providers-2.21.0.pom (3 KB at 223.1 KB/sec) [INFO] Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit4/2.21.0/surefire-junit4-2.21.0.jar [INFO] Downloaded: https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire-junit4/2.21.0/surefire-junit4-2.21.0.jar (83 KB at 2175.2 KB/sec) [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.couchbase.analytics.test.cbserver.ClusterExecutionITJava10 2018-08-12 14:08:55,820 main INFO Registered Log4j as the java.util.logging.LogManager. 2018-08-12T14:08:56.277-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [127.0.0.1:9000] 2018-08-12T14:08:56.290-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 10 seconds... 2018-08-12T14:08:56.295-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:08:58.363-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 0: auth: unauthorized-create-bucket] START 2018-08-12T14:08:58.382-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:09:08.390-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:09:08.429-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:09:08.430-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:09:08.431-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:09:14.863-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:09:14.875-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:09:15.486-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:09:15.504-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:09:15.508-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-1-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:09:31.552-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-create-bucket ... 2018-08-12T14:09:31.609-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:09:31.622-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:09:31.656-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:09:32.179-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-create-bucket/unauthorized-create-bucket.1.ddl.sqlpp raised an (expected) exception 2018-08-12T14:09:32.179-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-create-bucket PASSED 2018-08-12T14:09:32.182-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-create-bucket... 2018-08-12T14:09:33.001-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:09:34.386-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:09:34.387-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-create-bucket 2018-08-12T14:09:34.387-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:09:34.459-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:09:34.999-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:09:35.008-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:09:35.008-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:09:37.040-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 0: auth: unauthorized-create-bucket]'... 2018-08-12T14:09:37.133-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 0: auth: unauthorized-create-bucket] SUCCEEDED 2018-08-12T14:09:37.135-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 1: auth: unauthorized-connect-bucket] START 2018-08-12T14:09:37.179-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:09:49.601-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:09:49.624-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:09:49.625-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:09:49.625-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:09:54.489-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:09:54.493-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:09:54.836-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:09:54.841-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:09:54.842-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-2-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:10:09.556-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-connect-bucket ... 2018-08-12T14:10:09.595-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:10:09.606-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:10:09.635-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:10:10.578-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-connect-bucket/unauthorized-connect-bucket.3.ddl.sqlpp raised an (expected) exception 2018-08-12T14:10:10.578-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-connect-bucket PASSED 2018-08-12T14:10:10.579-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-connect-bucket... 2018-08-12T14:10:11.549-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:10:11.878-07:00 INFO TestExecutor [main] Dropping dataset gbook_users 2018-08-12T14:10:12.893-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:10:12.893-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-connect-bucket 2018-08-12T14:10:12.893-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:10:12.899-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:10:13.545-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:10:13.553-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:10:13.553-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:10:15.599-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 1: auth: unauthorized-connect-bucket]'... 2018-08-12T14:10:15.667-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 1: auth: unauthorized-connect-bucket] SUCCEEDED 2018-08-12T14:10:15.668-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 2: auth: unauthorized-disconnect-bucket] START 2018-08-12T14:10:15.695-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:10:26.772-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:10:26.780-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:10:26.781-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:10:26.781-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:10:30.362-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:10:30.365-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:10:30.688-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:10:30.693-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:10:30.694-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-3-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:10:45.583-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-disconnect-bucket ... 2018-08-12T14:10:45.592-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:10:45.600-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:10:45.617-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:10:49.770-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-disconnect-bucket/unauthorized-disconnect-bucket.3.ddl.sqlpp raised an (expected) exception 2018-08-12T14:10:49.771-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-disconnect-bucket PASSED 2018-08-12T14:10:49.771-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-disconnect-bucket... 2018-08-12T14:10:50.233-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:10:51.626-07:00 INFO TestExecutor [main] Dropping dataset gbook_user_ds 2018-08-12T14:10:52.510-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:10:52.511-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-disconnect-bucket 2018-08-12T14:10:52.511-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:10:52.516-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:10:53.153-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:10:53.160-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:10:53.160-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:10:55.174-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 2: auth: unauthorized-disconnect-bucket]'... 2018-08-12T14:10:55.245-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 2: auth: unauthorized-disconnect-bucket] SUCCEEDED 2018-08-12T14:10:55.248-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 3: auth: unauthorized-drop-bucket] START 2018-08-12T14:10:55.284-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:11:07.387-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:11:07.412-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:11:07.413-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:11:07.413-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:11:12.390-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:11:12.393-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:11:12.668-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:11:12.672-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:11:12.672-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-4-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:11:27.324-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-drop-bucket ... 2018-08-12T14:11:27.356-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:11:27.364-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:11:27.389-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:11:28.338-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-drop-bucket/unauthorized-drop-bucket.3.ddl.sqlpp raised an (expected) exception 2018-08-12T14:11:28.339-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-drop-bucket PASSED 2018-08-12T14:11:28.339-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-drop-bucket... 2018-08-12T14:11:29.437-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:11:29.820-07:00 INFO TestExecutor [main] Dropping dataset gbook_users 2018-08-12T14:11:30.743-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:11:30.743-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-drop-bucket 2018-08-12T14:11:30.743-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:11:30.754-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:11:31.352-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:11:31.359-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:11:31.360-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:11:33.372-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 3: auth: unauthorized-drop-bucket]'... 2018-08-12T14:11:33.439-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 3: auth: unauthorized-drop-bucket] SUCCEEDED 2018-08-12T14:11:33.440-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 4: auth: missing-bucket-read] START 2018-08-12T14:11:33.475-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:11:44.804-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:11:44.840-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:11:44.841-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:11:44.841-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:11:49.223-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:11:49.228-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:11:49.584-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:11:49.588-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:11:49.589-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-5-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:12:04.922-07:00 INFO TestExecutor [main] Starting [TEST]: auth/missing-bucket-read ... 2018-08-12T14:12:04.955-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:12:04.959-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:12:04.967-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:12:05.460-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/missing-bucket-read/missing-bucket-read.1.ddl.sqlpp raised an (expected) exception 2018-08-12T14:12:05.460-07:00 INFO TestExecutor [main] [TEST]: auth/missing-bucket-read PASSED 2018-08-12T14:12:05.460-07:00 INFO TestExecutor [main] Cleaning up after test case auth: missing-bucket-read... 2018-08-12T14:12:06.248-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:12:07.465-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:12:07.465-07:00 INFO TestExecutor [main] Clean up completed for test case auth: missing-bucket-read 2018-08-12T14:12:07.465-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:12:07.470-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:12:08.069-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:12:08.077-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:12:08.077-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:12:10.085-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 4: auth: missing-bucket-read]'... 2018-08-12T14:12:10.124-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 4: auth: missing-bucket-read] SUCCEEDED 2018-08-12T14:12:10.125-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 5: auth: unauthorized-create-shadow] START 2018-08-12T14:12:10.147-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:12:21.210-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:12:21.226-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:12:21.226-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:12:21.226-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:12:25.919-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:12:25.924-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:12:26.183-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:12:26.190-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:12:26.191-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-6-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:12:41.504-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-create-shadow ... 2018-08-12T14:12:41.524-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:12:41.529-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:12:41.559-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:12:42.066-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-create-shadow/unauthorized-create-shadow.3.ddl.sqlpp raised an (expected) exception 2018-08-12T14:12:42.066-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-create-shadow PASSED 2018-08-12T14:12:42.066-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-create-shadow... 2018-08-12T14:12:43.049-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:12:44.299-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:12:44.300-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-create-shadow 2018-08-12T14:12:44.300-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:12:44.322-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:12:44.837-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:12:44.845-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:12:44.845-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:12:46.859-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 5: auth: unauthorized-create-shadow]'... 2018-08-12T14:12:46.910-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 5: auth: unauthorized-create-shadow] SUCCEEDED 2018-08-12T14:12:46.910-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 6: auth: unauthorized-drop-shadow] START 2018-08-12T14:12:46.931-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:12:59.002-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:12:59.014-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:12:59.015-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:12:59.015-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:13:03.958-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:13:03.961-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:13:04.296-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:13:04.299-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:13:04.299-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-7-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:13:19.527-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-drop-shadow ... 2018-08-12T14:13:19.536-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:13:19.543-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:13:19.571-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:13:20.408-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-drop-shadow/unauthorized-drop-shadow.3.ddl.sqlpp raised an (expected) exception 2018-08-12T14:13:20.408-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-drop-shadow PASSED 2018-08-12T14:13:20.408-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-drop-shadow... 2018-08-12T14:13:21.154-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:13:21.554-07:00 INFO TestExecutor [main] Dropping dataset gbook_user_ds 2018-08-12T14:13:22.494-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:13:22.494-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-drop-shadow 2018-08-12T14:13:22.494-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:13:22.499-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:13:23.103-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:13:23.112-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:13:23.112-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:13:25.129-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 6: auth: unauthorized-drop-shadow]'... 2018-08-12T14:13:25.173-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 6: auth: unauthorized-drop-shadow] SUCCEEDED 2018-08-12T14:13:25.174-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 7: auth: unauthorized-create-index] START 2018-08-12T14:13:25.201-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:13:37.016-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:13:37.065-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:13:37.065-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:13:37.066-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:13:42.702-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:13:42.706-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:13:43.018-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:13:43.022-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:13:43.023-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-8-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:13:58.412-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-create-index ... 2018-08-12T14:13:58.420-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:13:58.433-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:13:58.447-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:13:59.359-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-create-index/unauthorized-create-index.3.ddl.sqlpp raised an (expected) exception 2018-08-12T14:13:59.360-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-create-index PASSED 2018-08-12T14:13:59.360-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-create-index... 2018-08-12T14:14:00.194-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:14:00.610-07:00 INFO TestExecutor [main] Dropping dataset gbook_user_ds 2018-08-12T14:14:01.690-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:14:01.690-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-create-index 2018-08-12T14:14:01.691-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:14:01.717-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:14:02.306-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:14:02.315-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:14:02.315-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:14:04.329-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 7: auth: unauthorized-create-index]'... 2018-08-12T14:14:04.388-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 7: auth: unauthorized-create-index] SUCCEEDED 2018-08-12T14:14:04.389-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 8: auth: unauthorized-drop-index] START 2018-08-12T14:14:04.429-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:14:18.173-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:14:18.194-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:14:18.195-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:14:18.195-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:14:23.784-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:14:23.788-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:14:24.129-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:14:24.138-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:14:24.138-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-9-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:14:40.437-07:00 INFO TestExecutor [main] Starting [TEST]: auth/unauthorized-drop-index ... 2018-08-12T14:14:40.473-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:14:40.531-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:14:40.550-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:14:41.921-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/auth/unauthorized-drop-index/unauthorized-drop-index.3.ddl.sqlpp raised an (expected) exception 2018-08-12T14:14:41.921-07:00 INFO TestExecutor [main] [TEST]: auth/unauthorized-drop-index PASSED 2018-08-12T14:14:41.921-07:00 INFO TestExecutor [main] Cleaning up after test case auth: unauthorized-drop-index... 2018-08-12T14:14:42.662-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:14:43.030-07:00 INFO TestExecutor [main] Dropping dataset gbook_user_ds 2018-08-12T14:14:43.968-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:14:43.968-07:00 INFO TestExecutor [main] Clean up completed for test case auth: unauthorized-drop-index 2018-08-12T14:14:43.968-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:14:43.990-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:14:44.606-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:14:44.614-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:14:44.614-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:14:46.622-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 8: auth: unauthorized-drop-index]'... 2018-08-12T14:14:46.680-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 8: auth: unauthorized-drop-index] SUCCEEDED 2018-08-12T14:14:46.681-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 9: rollback: no-rollback-on-double-kill-memcached] START 2018-08-12T14:14:46.711-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:14:59.075-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:14:59.133-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:14:59.133-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:14:59.133-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:15:03.912-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:15:03.915-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:15:04.257-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:15:04.263-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:15:04.263-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-10-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:15:19.213-07:00 INFO TestExecutor [main] Starting [TEST]: rollback/no-rollback-on-double-kill-memcached ... 2018-08-12T14:15:19.239-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:15:19.246-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:15:19.262-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:15:23.692-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json 2018-08-12T14:15:25.767-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:15:25.776-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:15:25.776-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json 2018-08-12T14:15:29.435-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:15:29.438-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:15:29.438-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json 2018-08-12T14:15:32.726-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:15:32.732-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:15:32.733-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json 2018-08-12T14:15:36.431-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:15:36.436-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:15:37.597-07:00 INFO TestExecutor [pool-73-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/no-rollback-on-double-kill-memcached/no-rollback-on-double-kill-memcached.005.adm 2018-08-12T14:15:37.632-07:00 INFO TestExecutor [main] Bucket gbook_users has been persisted 2018-08-12T14:15:37.647-07:00 INFO KillAllCommand [main] Executing: sh -c pkill -9 memcached 2018-08-12T14:15:37.695-07:00 INFO TestExecutor [main] KillAllCommand exit code: 0 2018-08-12T14:15:43.251-07:00 INFO KillAllCommand [main] Executing: sh -c pkill -9 memcached 2018-08-12T14:15:43.278-07:00 INFO TestExecutor [main] KillAllCommand exit code: 0 2018-08-12T14:15:48.351-07:00 INFO TestExecutor [main] Expected results file: target/rttest/rollback/no-rollback-on-double-kill-memcached.11.adm 2018-08-12T14:15:49.777-07:00 INFO TestExecutor [main] [TEST]: rollback/no-rollback-on-double-kill-memcached PASSED 2018-08-12T14:15:49.777-07:00 INFO TestExecutor [main] Cleaning up after test case rollback: no-rollback-on-double-kill-memcached... 2018-08-12T14:15:49.861-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:15:50.881-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:15:50.882-07:00 INFO TestExecutor [main] Clean up completed for test case rollback: no-rollback-on-double-kill-memcached 2018-08-12T14:15:50.882-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:15:50.889-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:15:51.621-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:15:51.630-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:15:51.630-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:15:53.640-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 9: rollback: no-rollback-on-double-kill-memcached]'... 2018-08-12T14:15:53.693-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 9: rollback: no-rollback-on-double-kill-memcached] SUCCEEDED 2018-08-12T14:15:53.694-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 10: rollback: partial-rollback-primary-only] START 2018-08-12T14:15:53.728-07:00 INFO TestExecutor [main] Starting [TEST]: rollback/partial-rollback-primary-only ... 2018-08-12T14:15:53.729-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:16:06.078-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:16:06.107-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:16:06.107-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:16:06.107-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/analytics_datadir/target/iodevicestest/iodevices/1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/analytics_datadir/target/iodevicestest/iodevices/2] 2018-08-12T14:16:11.342-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:16:11.346-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:16:11.718-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:16:11.721-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:16:11.722-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-11-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:16:29.073-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:16:29.089-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:16:29.121-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:16:33.476-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json 2018-08-12T14:16:35.182-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:35.186-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:35.186-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json 2018-08-12T14:16:38.250-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:38.256-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:38.257-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json 2018-08-12T14:16:41.021-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:41.025-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:41.026-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json 2018-08-12T14:16:44.391-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:44.395-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:44.396-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p5/gbook_users.json 2018-08-12T14:16:47.374-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p5/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:47.381-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:47.382-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p6/gbook_users.json 2018-08-12T14:16:49.921-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p6/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:49.924-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:49.926-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p7/gbook_users.json 2018-08-12T14:16:52.284-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p7/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:52.287-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:52.288-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p8/gbook_users.json 2018-08-12T14:16:55.625-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p8/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:55.628-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:55.628-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p9/gbook_users.json 2018-08-12T14:16:58.099-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p9/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:16:58.103-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:16:59.041-07:00 INFO TestExecutor [pool-88-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.005.adm 2018-08-12T14:16:59.087-07:00 INFO TestExecutor [main] Bucket gbook_users has been persisted 2018-08-12T14:16:59.110-07:00 INFO CBStopPersistenceCommand [main] Executing: sh -c ./cbepctl 127.0.0.1:12000 stop -u couchbase -p couchbase -b gbook_users 2018-08-12T14:17:02.264-07:00 INFO ClusterExecutionITBase [main+] >> Persistence stopped 2018-08-12T14:17:02.266-07:00 INFO TestExecutor [main] CBStopPersistenceCommand exit code: 0 2018-08-12T14:17:02.685-07:00 INFO TestExecutor [pool-90-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.009.adm 2018-08-12T14:17:04.482-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.012.adm 2018-08-12T14:17:04.483-07:00 INFO KillAllCommand [main] Executing: sh -c pkill -9 memcached 2018-08-12T14:17:04.520-07:00 INFO TestExecutor [main] KillAllCommand exit code: 0 2018-08-12T14:17:08.969-07:00 INFO TestExecutor [pool-96-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.015.adm 2018-08-12T14:17:08.970-07:00 INFO TestExecutor [pool-96-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.015.adm 2018-08-12T14:17:08.972-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-primary-only/partial-rollback-primary-only.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 62866 } 2018-08-12T14:17:10.558-07:00 INFO TestExecutor [pool-96-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.015.adm 2018-08-12T14:17:10.558-07:00 INFO TestExecutor [pool-96-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.015.adm 2018-08-12T14:17:10.558-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-primary-only/partial-rollback-primary-only.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 72134 } 2018-08-12T14:17:11.894-07:00 INFO TestExecutor [pool-96-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.015.adm 2018-08-12T14:17:11.895-07:00 INFO TestExecutor [pool-96-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.015.adm 2018-08-12T14:17:11.895-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-primary-only/partial-rollback-primary-only.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 83931 } 2018-08-12T14:17:13.205-07:00 INFO TestExecutor [pool-96-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-primary-only/partial-rollback-primary-only.015.adm 2018-08-12T14:17:13.330-07:00 INFO TestExecutor [main] Expected results file: target/rttest/rollback/partial-rollback-primary-only.10.adm 2018-08-12T14:17:14.743-07:00 INFO TestExecutor [main] [TEST]: rollback/partial-rollback-primary-only PASSED 2018-08-12T14:17:14.743-07:00 INFO TestExecutor [main] Cleaning up after test case rollback: partial-rollback-primary-only... 2018-08-12T14:17:14.816-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:17:15.732-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:17:15.732-07:00 INFO TestExecutor [main] Clean up completed for test case rollback: partial-rollback-primary-only 2018-08-12T14:17:15.732-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:17:15.735-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:17:16.382-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:17:16.391-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:17:16.391-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:17:18.402-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 10: rollback: partial-rollback-primary-only]'... 2018-08-12T14:17:18.464-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 10: rollback: partial-rollback-primary-only] SUCCEEDED 2018-08-12T14:17:18.465-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 11: rollback: partial-rollback-with-secondaries] START 2018-08-12T14:17:18.502-07:00 INFO TestExecutor [main] Starting [TEST]: rollback/partial-rollback-with-secondaries ... 2018-08-12T14:17:18.503-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:17:31.456-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:17:31.502-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:17:31.502-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:17:31.502-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/analytics_datadir/target/iodevicestest/iodevices/1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/analytics_datadir/target/iodevicestest/iodevices/2] 2018-08-12T14:17:36.668-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:17:36.672-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:17:36.979-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:17:36.982-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:17:36.982-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-12-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:17:52.784-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:17:52.799-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:17:52.830-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:17:56.806-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json 2018-08-12T14:17:58.553-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:17:58.582-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:17:58.583-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json 2018-08-12T14:18:00.654-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:00.656-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:00.657-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json 2018-08-12T14:18:03.210-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:03.213-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:03.213-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json 2018-08-12T14:18:05.342-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:05.346-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:05.347-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p5/gbook_users.json 2018-08-12T14:18:08.168-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p5/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:08.173-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:08.173-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p6/gbook_users.json 2018-08-12T14:18:10.491-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p6/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:10.499-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:10.500-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p7/gbook_users.json 2018-08-12T14:18:13.536-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p7/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:13.540-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:13.541-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p8/gbook_users.json 2018-08-12T14:18:16.187-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p8/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:16.191-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:16.191-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p9/gbook_users.json 2018-08-12T14:18:18.406-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p9/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:18:18.409-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:18:19.602-07:00 INFO TestExecutor [pool-112-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.005.adm 2018-08-12T14:18:19.604-07:00 INFO TestExecutor [pool-112-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.005.adm 2018-08-12T14:18:19.605-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.005.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 74061 } 2018-08-12T14:18:21.232-07:00 INFO TestExecutor [pool-112-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.005.adm 2018-08-12T14:18:21.232-07:00 INFO TestExecutor [pool-112-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.005.adm 2018-08-12T14:18:21.233-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.005.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 89404 } 2018-08-12T14:18:22.575-07:00 INFO TestExecutor [pool-112-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.005.adm 2018-08-12T14:18:22.595-07:00 INFO TestExecutor [main] Bucket gbook_users has been persisted 2018-08-12T14:18:22.606-07:00 INFO CBStopPersistenceCommand [main] Executing: sh -c ./cbepctl 127.0.0.1:12000 stop -u couchbase -p couchbase -b gbook_users 2018-08-12T14:18:24.744-07:00 INFO ClusterExecutionITBase [main+] >> Persistence stopped 2018-08-12T14:18:24.746-07:00 INFO TestExecutor [main] CBStopPersistenceCommand exit code: 0 2018-08-12T14:18:25.073-07:00 INFO TestExecutor [pool-116-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.009.adm 2018-08-12T14:18:27.168-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.012.adm 2018-08-12T14:18:27.169-07:00 INFO KillAllCommand [main] Executing: sh -c pkill -9 memcached 2018-08-12T14:18:27.194-07:00 INFO TestExecutor [main] KillAllCommand exit code: 0 2018-08-12T14:18:31.706-07:00 INFO TestExecutor [pool-122-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.015.adm 2018-08-12T14:18:31.707-07:00 INFO TestExecutor [pool-122-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.015.adm 2018-08-12T14:18:31.707-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 57343 } 2018-08-12T14:18:33.049-07:00 INFO TestExecutor [pool-122-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.015.adm 2018-08-12T14:18:33.049-07:00 INFO TestExecutor [pool-122-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.015.adm 2018-08-12T14:18:33.049-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 65744 } 2018-08-12T14:18:34.408-07:00 INFO TestExecutor [pool-122-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.015.adm 2018-08-12T14:18:34.408-07:00 INFO TestExecutor [pool-122-thread-1] Actual results file: target/rttest/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.015.adm 2018-08-12T14:18:34.409-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 79242 } 2018-08-12T14:18:35.713-07:00 INFO TestExecutor [pool-122-thread-1] Expected results file: src/test/resources/runtimets/results/rollback/partial-rollback-with-secondaries/partial-rollback-with-secondaries.015.adm 2018-08-12T14:18:35.789-07:00 INFO TestExecutor [main] Expected results file: target/rttest/rollback/partial-rollback-with-secondaries.10.adm 2018-08-12T14:18:37.301-07:00 INFO TestExecutor [main] [TEST]: rollback/partial-rollback-with-secondaries PASSED 2018-08-12T14:18:37.302-07:00 INFO TestExecutor [main] Cleaning up after test case rollback: partial-rollback-with-secondaries... 2018-08-12T14:18:37.391-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:18:38.415-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:18:38.415-07:00 INFO TestExecutor [main] Clean up completed for test case rollback: partial-rollback-with-secondaries 2018-08-12T14:18:38.415-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:18:38.430-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:18:39.071-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:18:39.078-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:18:39.079-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:18:41.086-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 11: rollback: partial-rollback-with-secondaries]'... 2018-08-12T14:18:41.136-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 11: rollback: partial-rollback-with-secondaries] SUCCEEDED 2018-08-12T14:18:41.137-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 12: lifecycle: kill_nc] START 2018-08-12T14:18:41.162-07:00 INFO TestExecutor [main] Starting [TEST]: lifecycle/kill_nc ... 2018-08-12T14:18:41.162-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:18:51.878-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:18:51.917-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:18:51.917-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:18:51.918-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:18:56.986-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:18:56.989-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:18:57.307-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:18:57.312-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1 2018-08-12T14:18:57.313-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-13-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:19:17.393-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_nc/kill_nc.3.adm 2018-08-12T14:19:17.395-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1, 2] 2018-08-12T14:19:17.424-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1, 2]]) are fully removed 2018-08-12T14:19:17.446-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:19:17.448-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1, 2] with service(s): [ANALYTICS] 2018-08-12T14:19:21.485-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:19:21.761-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:19:21.765-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9001,127.0.0.1:9002, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:19:30.763-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:19:30.768-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:19:52.210-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:19:52.210-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:19:52.214-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 3 2018-08-12T14:19:52.215-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-14-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:19:52.215-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-14-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:19:52.215-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-14-/127.0.0.1:9600] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:19:52.699-07:00 INFO ClusterExecutionITBase [main] Killing node 2 (signalNum: 15) 2018-08-12T14:19:53.114-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:19:56.235-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:19:57.138-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:19:58.157-07:00 ERRO asterix [pool-148-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:19:58.188-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:19:59.742-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:20:03.577-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:20:04.596-07:00 ERRO asterix [pool-151-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:20:04.615-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:20:05.659-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:20:06.773-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:20:09.563-07:00 INFO TestExecutor [pool-146-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_nc/kill_nc.6.adm 2018-08-12T14:20:09.567-07:00 INFO ClusterExecutionITBase [main] Killing node 2 (signalNum: 9) 2018-08-12T14:20:10.713-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:20:12.254-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:20:13.268-07:00 ERRO asterix [pool-158-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:20:13.278-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:20:15.790-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:20:16.872-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:20:17.885-07:00 ERRO asterix [pool-161-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:20:17.888-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:20:18.923-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:20:21.348-07:00 INFO TestExecutor [pool-155-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_nc/kill_nc.8.adm 2018-08-12T14:20:21.349-07:00 INFO TestExecutor [main] [TEST]: lifecycle/kill_nc PASSED 2018-08-12T14:20:21.349-07:00 INFO TestExecutor [main] Cleaning up after test case lifecycle: kill_nc... 2018-08-12T14:20:22.116-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:20:22.155-07:00 INFO TestExecutor [main] Disconnecting link tpch1.Local 2018-08-12T14:20:22.247-07:00 INFO TestExecutor [main] Disconnecting link tpch2.Local 2018-08-12T14:20:23.036-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: tpch1,tpch2 2018-08-12T14:20:23.448-07:00 INFO TestExecutor [main] Clean up completed for test case lifecycle: kill_nc 2018-08-12T14:20:23.448-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:20:23.477-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:20:24.183-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:20:24.191-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:20:24.192-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:20:26.211-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 12: lifecycle: kill_nc]'... 2018-08-12T14:20:26.301-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 12: lifecycle: kill_nc] SUCCEEDED 2018-08-12T14:20:26.302-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 13: lifecycle: kill_cbas] START 2018-08-12T14:20:26.355-07:00 INFO TestExecutor [main] Starting [TEST]: lifecycle/kill_cbas ... 2018-08-12T14:20:26.356-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:20:38.413-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:20:38.434-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:20:38.434-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:20:38.435-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:20:43.497-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:20:43.499-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:20:43.783-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:20:43.788-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:20:43.788-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-15-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:21:01.468-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cbas/kill_cbas.3.adm 2018-08-12T14:21:01.469-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1, 2] 2018-08-12T14:21:01.482-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1, 2]]) are fully removed 2018-08-12T14:21:01.490-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:21:01.490-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1, 2] with service(s): [ANALYTICS] 2018-08-12T14:21:05.531-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:21:05.898-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:21:05.901-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9002,127.0.0.1:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:21:13.771-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:21:13.780-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:21:33.209-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:21:33.209-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:21:33.213-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 3 2018-08-12T14:21:33.213-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-16-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:21:33.217-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-16-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:21:33.217-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-16-/127.0.0.1:9600] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:21:36.243-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cbas/kill_cbas.5.adm 2018-08-12T14:21:36.245-07:00 INFO ClusterExecutionITBase [main] Killing 'cbas' wrapper for node 2 (signalNum: 15) 2018-08-12T14:21:36.968-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:21:37.495-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:21:37.504-07:00 ERRO asterix [pool-186-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:21:37.506-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:21:39.258-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:21:40.984-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:21:41.996-07:00 ERRO asterix [pool-189-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:21:41.999-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:21:45.244-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:21:46.386-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:21:48.030-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:21:49.073-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:21:51.274-07:00 INFO TestExecutor [pool-185-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cbas/kill_cbas.7.adm 2018-08-12T14:21:51.276-07:00 INFO ClusterExecutionITBase [main] Killing 'cbas' wrapper for node 2 (signalNum: 9) 2018-08-12T14:21:52.683-07:00 ERRO asterix [pool-196-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.NoHttpResponseException: localhost:9630 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] 2018-08-12T14:21:52.686-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.NoHttpResponseException: localhost:9630 failed to respond 2018-08-12T14:21:54.231-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:21:55.869-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:21:56.874-07:00 ERRO asterix [pool-199-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:21:56.878-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:21:58.571-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:22:00.716-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:22:01.738-07:00 ERRO asterix [pool-202-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:22:01.743-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:22:02.782-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:22:03.825-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:22:06.547-07:00 INFO TestExecutor [pool-195-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cbas/kill_cbas.9.adm 2018-08-12T14:22:06.548-07:00 INFO TestExecutor [main] [TEST]: lifecycle/kill_cbas PASSED 2018-08-12T14:22:06.549-07:00 INFO TestExecutor [main] Cleaning up after test case lifecycle: kill_cbas... 2018-08-12T14:22:06.626-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:22:06.725-07:00 INFO TestExecutor [main] Disconnecting link tpch1.Local 2018-08-12T14:22:07.444-07:00 INFO TestExecutor [main] Disconnecting link tpch2.Local 2018-08-12T14:22:07.777-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: tpch1,tpch2 2018-08-12T14:22:08.203-07:00 INFO TestExecutor [main] Clean up completed for test case lifecycle: kill_cbas 2018-08-12T14:22:08.204-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:22:08.219-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:22:08.900-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:22:08.907-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:22:08.908-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:22:10.913-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 13: lifecycle: kill_cbas]'... 2018-08-12T14:22:11.004-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 13: lifecycle: kill_cbas] SUCCEEDED 2018-08-12T14:22:11.005-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 14: lifecycle: kill_cc] START 2018-08-12T14:22:11.048-07:00 INFO TestExecutor [main] Starting [TEST]: lifecycle/kill_cc ... 2018-08-12T14:22:11.050-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:22:23.203-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:22:23.231-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:22:23.231-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:22:23.232-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:22:28.307-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:22:28.310-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:22:28.586-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:22:28.589-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:22:28.589-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-17-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:22:47.235-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cc/kill_cc.3.adm 2018-08-12T14:22:47.237-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1, 2] 2018-08-12T14:22:47.248-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1, 2]]) are fully removed 2018-08-12T14:22:47.268-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:22:47.268-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1, 2] with service(s): [ANALYTICS] 2018-08-12T14:22:51.481-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:22:51.679-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:22:51.684-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9001,127.0.0.1:9002, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:23:00.269-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:23:00.274-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:23:19.671-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:23:19.672-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:23:19.676-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 3 2018-08-12T14:23:19.677-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-18-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:23:19.677-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-18-/127.0.0.1:9600] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:23:19.677-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-18-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:23:21.864-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cc/kill_cc.5.adm 2018-08-12T14:23:21.865-07:00 INFO ClusterExecutionITBase [main] Killing node 0 (signalNum: 15) 2018-08-12T14:23:22.662-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:23:25.772-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:23:25.773-07:00 INFO TestExecutor [main] Waiting for cluster state .*... 2018-08-12T14:23:35.324-07:00 INFO TestExecutor [main] Cluster state now .* 2018-08-12T14:23:35.542-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:23:36.556-07:00 ERRO asterix [pool-229-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:23:36.559-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:23:37.752-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:23:38.830-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:23:40.121-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:23:41.204-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:23:45.194-07:00 INFO TestExecutor [pool-227-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cc/kill_cc.8.adm 2018-08-12T14:23:45.198-07:00 INFO ClusterExecutionITBase [main] Killing node 0 (signalNum: 9) 2018-08-12T14:23:45.984-07:00 INFO TestExecutor [main] Waiting for cluster state .*... 2018-08-12T14:23:46.035-07:00 INFO RetryExec [Thread-340] I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:9600: Connection reset 2018-08-12T14:23:46.035-07:00 INFO RetryExec [Thread-340] Retrying request to {}->http://localhost:9600 2018-08-12T14:23:52.543-07:00 INFO TestExecutor [main] Cluster state now .* 2018-08-12T14:23:52.562-07:00 ERRO asterix [pool-236-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:23:52.565-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:23:53.680-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:23:54.751-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:23:56.301-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:24:00.653-07:00 INFO TestExecutor [pool-235-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/kill_cc/kill_cc.12.adm 2018-08-12T14:24:00.655-07:00 INFO TestExecutor [main] [TEST]: lifecycle/kill_cc PASSED 2018-08-12T14:24:00.655-07:00 INFO TestExecutor [main] Cleaning up after test case lifecycle: kill_cc... 2018-08-12T14:24:00.960-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:24:01.068-07:00 INFO TestExecutor [main] Disconnecting link Metadata.Local 2018-08-12T14:24:01.164-07:00 INFO TestExecutor [main] Disconnecting link tpch1.Local 2018-08-12T14:24:01.240-07:00 INFO TestExecutor [main] Disconnecting link tpch2.Local 2018-08-12T14:24:02.138-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: tpch1,tpch2 2018-08-12T14:24:02.878-07:00 INFO TestExecutor [main] Clean up completed for test case lifecycle: kill_cc 2018-08-12T14:24:02.879-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:24:02.915-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:24:03.643-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:24:03.652-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:24:03.653-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:24:05.660-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 14: lifecycle: kill_cc]'... 2018-08-12T14:24:05.772-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 14: lifecycle: kill_cc] SUCCEEDED 2018-08-12T14:24:05.774-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 15: lifecycle: restart-node-api] START 2018-08-12T14:24:05.837-07:00 INFO TestExecutor [main] Starting [TEST]: lifecycle/restart-node-api ... 2018-08-12T14:24:05.838-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:24:16.119-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:24:16.140-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:24:16.140-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:24:16.140-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:24:21.380-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:24:21.384-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:24:21.643-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:24:21.647-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:24:21.647-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-19-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:24:36.736-07:00 INFO CBDocLoaderCommand [main] Executing: sh -c ./bin/cbdocloader -c 127.0.0.1:9000 -u couchbase -p couchbase -b beer-sample -m 100 -d ./samples/beer-sample.zip 2018-08-12T14:24:45.572-07:00 INFO ClusterExecutionITBase [main+] >> Errors occurred during the index creation phase. See logs for details. 2018-08-12T14:24:45.576-07:00 INFO TestExecutor [main] cbdocloader exit code 1 2018-08-12T14:24:45.576-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1] 2018-08-12T14:24:45.599-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1]]) are fully removed 2018-08-12T14:24:45.606-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:24:45.606-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1] with service(s): [ANALYTICS] 2018-08-12T14:24:50.645-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:24:50.648-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:24:54.943-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:24:54.947-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:25:09.351-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:25:09.351-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:25:09.356-07:00 INFO ClusterExecutionITBase [main] state: [0, 1], nodes: {{DATA,ANALYTICS},{ANALYTICS},{}}, cbas node count: 2 2018-08-12T14:25:09.356-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-20-/127.0.0.1:9615] Waiting for NCs == 2 & Cluster ACTIVE for up to 60s... 2018-08-12T14:25:09.357-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-20-/127.0.0.1:9600] Waiting for NCs == 2 & Cluster ACTIVE for up to 60s... 2018-08-12T14:25:17.371-07:00 INFO TestExecutor [pool-256-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.03.adm 2018-08-12T14:25:17.374-07:00 INFO TestExecutor [pool-256-thread-1] Actual results file: target/rttest/results/lifecycle/restart-node-api/restart-node-api.03.adm 2018-08-12T14:25:17.375-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/lifecycle/restart-node-api/restart-node-api.03.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 0 } 2018-08-12T14:25:18.958-07:00 INFO TestExecutor [pool-256-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.03.adm 2018-08-12T14:25:18.958-07:00 INFO TestExecutor [pool-256-thread-1] Actual results file: target/rttest/results/lifecycle/restart-node-api/restart-node-api.03.adm 2018-08-12T14:25:18.958-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/lifecycle/restart-node-api/restart-node-api.03.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 2886 } 2018-08-12T14:25:20.376-07:00 INFO TestExecutor [pool-256-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.03.adm 2018-08-12T14:25:20.377-07:00 INFO TestExecutor [pool-256-thread-1] Actual results file: target/rttest/results/lifecycle/restart-node-api/restart-node-api.03.adm 2018-08-12T14:25:20.377-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/lifecycle/restart-node-api/restart-node-api.03.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 2927 } 2018-08-12T14:25:21.578-07:00 INFO TestExecutor [pool-256-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.03.adm 2018-08-12T14:25:21.748-07:00 INFO TestExecutor [pool-261-thread-1] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.04.adm 2018-08-12T14:25:21.844-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.05.adm 2018-08-12T14:25:21.846-07:00 INFO TestExecutor [main] Waiting for cluster state ACTIVE... 2018-08-12T14:25:36.287-07:00 INFO RetryExec [Thread-350] I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://localhost:9615: The target server failed to respond 2018-08-12T14:25:36.288-07:00 INFO RetryExec [Thread-350] Retrying request to {}->http://localhost:9615 2018-08-12T14:25:49.356-07:00 INFO TestExecutor [main] Cluster state now ACTIVE 2018-08-12T14:25:50.201-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.07.adm 2018-08-12T14:25:50.429-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/lifecycle/restart-node-api/restart-node-api.08.adm 2018-08-12T14:25:53.524-07:00 INFO TestExecutor [main] [TEST]: lifecycle/restart-node-api PASSED 2018-08-12T14:25:53.524-07:00 INFO TestExecutor [main] Cleaning up after test case lifecycle: restart-node-api... 2018-08-12T14:25:53.990-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:25:54.039-07:00 INFO TestExecutor [main] Disconnecting link Metadata.Local 2018-08-12T14:25:54.483-07:00 INFO TestExecutor [main] Clean up completed for test case lifecycle: restart-node-api 2018-08-12T14:25:54.484-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:25:54.501-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:25:55.159-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:25:55.180-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:25:55.180-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:25:57.192-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 15: lifecycle: restart-node-api]'... 2018-08-12T14:25:57.425-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 15: lifecycle: restart-node-api] SUCCEEDED 2018-08-12T14:25:57.426-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 16: recovery: kill-nc-delete-kv-bucket-then-rebalance] START 2018-08-12T14:25:57.538-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:26:09.668-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:26:09.706-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:26:09.706-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:26:09.706-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:26:15.320-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:26:15.323-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:26:15.619-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:26:15.624-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:26:15.624-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-21-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:26:31.601-07:00 INFO TestExecutor [main] Starting [TEST]: recovery/kill-nc-delete-kv-bucket-then-rebalance ... 2018-08-12T14:26:31.602-07:00 INFO CBDocLoaderCommand [main] Executing: sh -c ./bin/cbdocloader -c 127.0.0.1:9000 -u couchbase -p couchbase -b beer-sample -m 100 -d ./samples/beer-sample.zip 2018-08-12T14:26:38.918-07:00 INFO ClusterExecutionITBase [main+] >> Errors occurred during the index creation phase. See logs for details. 2018-08-12T14:26:38.922-07:00 INFO TestExecutor [main] cbdocloader exit code 1 2018-08-12T14:26:38.924-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1, 2] 2018-08-12T14:26:38.936-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1, 2]]) are fully removed 2018-08-12T14:26:38.942-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:26:38.942-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1, 2] with service(s): [ANALYTICS] 2018-08-12T14:26:43.507-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:26:43.553-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:26:43.557-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9001,127.0.0.1:9002, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:26:53.032-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:26:53.036-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:27:09.448-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:27:09.448-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:27:09.453-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 3 2018-08-12T14:27:09.454-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-22-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:27:09.454-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-22-/127.0.0.1:9600] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:27:09.454-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-22-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:27:18.967-07:00 INFO TestExecutor [pool-281-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:18.968-07:00 INFO TestExecutor [pool-281-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:18.971-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.004.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 0 } 2018-08-12T14:27:20.872-07:00 INFO TestExecutor [pool-281-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:20.872-07:00 INFO TestExecutor [pool-281-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:20.876-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.004.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 1918 } 2018-08-12T14:27:22.802-07:00 INFO TestExecutor [pool-281-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:22.803-07:00 INFO TestExecutor [pool-281-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:22.803-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.004.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 1918 } 2018-08-12T14:27:24.158-07:00 INFO TestExecutor [pool-281-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:24.158-07:00 INFO TestExecutor [pool-281-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:24.159-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.004.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 5657 } 2018-08-12T14:27:25.371-07:00 INFO TestExecutor [pool-281-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.005.adm 2018-08-12T14:27:25.604-07:00 INFO TestExecutor [pool-287-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.006.adm 2018-08-12T14:27:25.606-07:00 INFO ClusterExecutionITBase [main] Killing node 2 (signalNum: 9) 2018-08-12T14:27:26.225-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:27:26.302-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:27:27.483-07:00 INFO KvStoreHttpHelper [main] Bucket (beer-sample) deleted successfully. 2018-08-12T14:27:27.515-07:00 ERRO asterix [pool-290-thread-1] Failure executing GET http://localhost:9630/analytics/buckets HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:27:27.521-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:27:28.567-07:00 INFO TestExecutor [pool-289-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:28.568-07:00 INFO TestExecutor [pool-289-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:28.569-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.pollget.http changed at line 6: expected < "state" : "disconnected" actual > "state" : "connected" 2018-08-12T14:27:29.606-07:00 INFO TestExecutor [pool-289-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:29.606-07:00 INFO TestExecutor [pool-289-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:29.607-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.pollget.http changed at line 6: expected < "state" : "disconnected" actual > "state" : "connected" 2018-08-12T14:27:30.612-07:00 ERRO asterix [pool-293-thread-1] Failure executing GET http://localhost:9630/analytics/buckets HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:27:30.614-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:27:31.653-07:00 INFO TestExecutor [pool-289-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:31.653-07:00 INFO TestExecutor [pool-289-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:31.654-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.pollget.http changed at line 2: expected < "buckets" : [ { actual > 2018-08-12T14:27:32.695-07:00 INFO TestExecutor [pool-289-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:32.695-07:00 INFO TestExecutor [pool-289-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:32.695-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.pollget.http changed at line 2: expected < "buckets" : [ { actual > 2018-08-12T14:27:33.873-07:00 INFO TestExecutor [pool-289-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:33.874-07:00 INFO TestExecutor [pool-289-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:33.874-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.pollget.http changed at line 2: expected < "buckets" : [ { actual > 2018-08-12T14:27:34.900-07:00 INFO TestExecutor [pool-289-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-delete-kv-bucket-then-rebalance/kill-nc-delete-kv-bucket-then-rebalance.009.adm 2018-08-12T14:27:36.770-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0, 1, 2] to [0] 2018-08-12T14:27:36.778-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([]) are fully removed 2018-08-12T14:27:36.779-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:27:36.779-07:00 INFO ClusterExecutionITBase [main] removing nodes [1, 2] 2018-08-12T14:27:36.781-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-remove, 127.0.0.1:9001,127.0.0.1:9002] 2018-08-12T14:27:41.030-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:27:41.031-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:27:41.036-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:27:41.036-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-23-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:27:41.183-07:00 INFO TestExecutor [main] Expected results file: target/rttest/recovery/kill-nc-delete-kv-bucket-then-rebalance.10.adm 2018-08-12T14:27:41.273-07:00 INFO TestExecutor [main] Expected results file: target/rttest/recovery/kill-nc-delete-kv-bucket-then-rebalance.11.adm 2018-08-12T14:27:41.273-07:00 INFO TestExecutor [main] [TEST]: recovery/kill-nc-delete-kv-bucket-then-rebalance PASSED 2018-08-12T14:27:41.273-07:00 INFO TestExecutor [main] Cleaning up after test case recovery: kill-nc-delete-kv-bucket-then-rebalance... 2018-08-12T14:27:41.355-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:27:41.551-07:00 INFO TestExecutor [main] Dropping dataset bandb 2018-08-12T14:27:41.652-07:00 INFO TestExecutor [main] Dropping dataset beers 2018-08-12T14:27:41.757-07:00 INFO TestExecutor [main] Dropping dataset breweries 2018-08-12T14:27:42.023-07:00 INFO TestExecutor [main] Clean up completed for test case recovery: kill-nc-delete-kv-bucket-then-rebalance 2018-08-12T14:27:42.023-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:27:42.047-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:27:42.818-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:27:42.827-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:27:42.827-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:27:44.847-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 16: recovery: kill-nc-delete-kv-bucket-then-rebalance]'... 2018-08-12T14:27:44.911-07:00 WARN ClusterExecutionITBase [main] Ignoring IOException copying /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/cluster_run.log to target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 16: recovery: kill-nc-delete-kv-bucket-then-rebalance]/logs (will retry in 1s) java.io.IOException: Failed to copy full contents from '/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/cluster_run.log' to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 16: recovery: kill-nc-delete-kv-bucket-then-rebalance]/logs/cluster_run.log' Expected length: 5840171 Actual: 5838977 at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1142) ~[commons-io-2.6.jar:2.6] at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076) ~[commons-io-2.6.jar:2.6] at org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:1000) ~[commons-io-2.6.jar:2.6] at org.apache.commons.io.FileUtils.copyFileToDirectory(FileUtils.java:961) ~[commons-io-2.6.jar:2.6] at com.couchbase.analytics.test.common.ClusterTestBase.safeCopyDir(ClusterTestBase.java:836) [cbas-server-1.0.0-SNAPSHOT-tests.jar:6.0.0-0000] at com.couchbase.analytics.test.cbserver.ClusterExecutionITBase.after(ClusterExecutionITBase.java:182) [test-classes/:?] at com.couchbase.analytics.test.cbserver.ClusterExecutionITJava10.after(ClusterExecutionITJava10.java:52) [test-classes/:?] at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.junit.runners.Suite.runChild(Suite.java:128) [junit-4.12.jar:4.12] at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) [surefire-booter-2.21.0.jar:2.21.0] 2018-08-12T14:27:45.995-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 16: recovery: kill-nc-delete-kv-bucket-then-rebalance] SUCCEEDED 2018-08-12T14:27:45.996-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 17: recovery: kill-nc] START 2018-08-12T14:27:46.031-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:27:59.097-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:27:59.111-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:27:59.111-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:27:59.111-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:28:04.486-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:28:04.490-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:28:04.750-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:28:04.755-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:28:04.755-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-24-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:28:21.032-07:00 INFO TestExecutor [main] Starting [TEST]: recovery/kill-nc ... 2018-08-12T14:28:21.033-07:00 INFO CBDocLoaderCommand [main] Executing: sh -c ./bin/cbdocloader -c 127.0.0.1:9000 -u couchbase -p couchbase -b beer-sample -m 100 -d ./samples/beer-sample.zip 2018-08-12T14:28:28.070-07:00 INFO ClusterExecutionITBase [main+] >> Errors occurred during the index creation phase. See logs for details. 2018-08-12T14:28:28.073-07:00 INFO TestExecutor [main] cbdocloader exit code 1 2018-08-12T14:28:28.074-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1, 2] 2018-08-12T14:28:28.087-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1, 2]]) are fully removed 2018-08-12T14:28:28.097-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:28:28.098-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1, 2] with service(s): [ANALYTICS] 2018-08-12T14:28:32.892-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:28:33.320-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:28:33.323-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9002,127.0.0.1:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:28:41.477-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:28:41.482-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:28:56.903-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:28:56.903-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:28:56.908-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 3 2018-08-12T14:28:56.908-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-25-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:28:56.908-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-25-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:28:56.909-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-25-/127.0.0.1:9600] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:29:08.536-07:00 INFO TestExecutor [pool-316-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:08.537-07:00 INFO TestExecutor [pool-316-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:08.538-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.005.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 0 } 2018-08-12T14:29:10.333-07:00 INFO TestExecutor [pool-316-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:10.333-07:00 INFO TestExecutor [pool-316-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:10.334-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.005.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 1918 } 2018-08-12T14:29:12.799-07:00 INFO TestExecutor [pool-316-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:12.800-07:00 INFO TestExecutor [pool-316-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:12.800-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.005.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 1942 } 2018-08-12T14:29:14.330-07:00 INFO TestExecutor [pool-316-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:14.331-07:00 INFO TestExecutor [pool-316-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:14.331-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.005.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 5665 } 2018-08-12T14:29:15.654-07:00 INFO TestExecutor [pool-316-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.003.adm 2018-08-12T14:29:15.929-07:00 INFO TestExecutor [pool-322-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.004.adm 2018-08-12T14:29:15.930-07:00 INFO ClusterExecutionITBase [main] Killing node 2 (signalNum: 9) 2018-08-12T14:29:17.299-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:29:17.599-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:29:19.909-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:29:20.935-07:00 ERRO asterix [pool-326-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:29:20.939-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:29:22.785-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:29:24.426-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Internal error 2018-08-12T14:29:25.431-07:00 ERRO asterix [pool-329-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:29:25.435-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:29:28.352-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:29:29.544-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:29:30.559-07:00 ERRO asterix [pool-332-thread-1] Failure executing POST http://localhost:9630/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:29:30.568-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9630 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:29:31.655-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:29:32.759-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:29:35.002-07:00 INFO TestExecutor [pool-324-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:35.003-07:00 INFO TestExecutor [pool-324-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:35.004-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.009.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 3956 } 2018-08-12T14:29:36.404-07:00 INFO TestExecutor [pool-324-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:36.404-07:00 INFO TestExecutor [pool-324-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:36.405-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.009.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 3956 } 2018-08-12T14:29:38.763-07:00 INFO TestExecutor [pool-324-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:38.763-07:00 INFO TestExecutor [pool-324-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:38.763-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.009.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 4170 } 2018-08-12T14:29:40.472-07:00 INFO TestExecutor [pool-324-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:40.473-07:00 INFO TestExecutor [pool-324-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:40.473-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.009.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 5245 } 2018-08-12T14:29:41.689-07:00 INFO TestExecutor [pool-324-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:41.689-07:00 INFO TestExecutor [pool-324-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:41.689-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.009.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 5248 } 2018-08-12T14:29:42.988-07:00 INFO TestExecutor [pool-324-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:42.988-07:00 INFO TestExecutor [pool-324-thread-1] Actual results file: target/rttest/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:42.988-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.009.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 5532 } 2018-08-12T14:29:44.391-07:00 INFO TestExecutor [pool-324-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.007.adm 2018-08-12T14:29:44.622-07:00 INFO TestExecutor [pool-342-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc/kill-nc.008.adm 2018-08-12T14:29:44.626-07:00 INFO KvUpserter [Thread-375] KvUpserter stopped. Successful upserts: 4308, Failed upserts: 0 2018-08-12T14:29:44.940-07:00 INFO TestExecutor [pool-344-thread-1] Expected results file: target/rttest/recovery/kill-nc.12.polled.adm 2018-08-12T14:29:44.941-07:00 INFO TestExecutor [pool-344-thread-1] Actual results file: target/rttest/recovery/kill-nc.12.adm 2018-08-12T14:29:44.941-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.012.polldynamic.sqlpp changed at line 1: expected < { "count": 11557 } actual > { "count": 11611 } 2018-08-12T14:29:46.192-07:00 INFO TestExecutor [pool-344-thread-1] Expected results file: target/rttest/recovery/kill-nc.12.polled.adm 2018-08-12T14:29:46.192-07:00 INFO TestExecutor [pool-344-thread-1] Actual results file: target/rttest/recovery/kill-nc.12.adm 2018-08-12T14:29:46.192-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.012.polldynamic.sqlpp changed at line 1: expected < { "count": 11557 } actual > { "count": 11611 } 2018-08-12T14:29:47.370-07:00 INFO TestExecutor [pool-344-thread-1] Expected results file: target/rttest/recovery/kill-nc.12.polled.adm 2018-08-12T14:29:47.370-07:00 INFO TestExecutor [pool-344-thread-1] Actual results file: target/rttest/recovery/kill-nc.12.adm 2018-08-12T14:29:47.370-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.012.polldynamic.sqlpp changed at line 1: expected < { "count": 11557 } actual > { "count": 11611 } 2018-08-12T14:29:48.600-07:00 INFO TestExecutor [pool-344-thread-1] Expected results file: target/rttest/recovery/kill-nc.12.polled.adm 2018-08-12T14:29:48.600-07:00 INFO TestExecutor [pool-344-thread-1] Actual results file: target/rttest/recovery/kill-nc.12.adm 2018-08-12T14:29:48.601-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc/kill-nc.012.polldynamic.sqlpp changed at line 1: expected < { "count": 11557 } actual > { "count": 11611 } 2018-08-12T14:29:49.801-07:00 INFO TestExecutor [pool-344-thread-1] Expected results file: target/rttest/recovery/kill-nc.12.polled.adm 2018-08-12T14:29:52.650-07:00 INFO TestExecutor [main] [TEST]: recovery/kill-nc PASSED 2018-08-12T14:29:52.651-07:00 INFO TestExecutor [main] Cleaning up after test case recovery: kill-nc... 2018-08-12T14:29:52.751-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:29:53.303-07:00 INFO TestExecutor [main] Clean up completed for test case recovery: kill-nc 2018-08-12T14:29:53.303-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:29:53.314-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:29:54.178-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:29:54.186-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:29:54.186-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:29:56.192-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 17: recovery: kill-nc]'... 2018-08-12T14:29:56.441-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 17: recovery: kill-nc] SUCCEEDED 2018-08-12T14:29:56.441-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 18: recovery: kill-cc-single-node] START 2018-08-12T14:29:56.569-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:30:07.248-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:30:07.268-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:30:07.268-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:30:07.268-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:30:12.549-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:30:12.552-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:30:12.794-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:30:12.799-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:30:12.799-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-26-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:30:29.278-07:00 INFO TestExecutor [main] Starting [TEST]: recovery/kill-cc-single-node ... 2018-08-12T14:30:29.279-07:00 INFO CBDocLoaderCommand [main] Executing: sh -c ./bin/cbdocloader -c 127.0.0.1:9000 -u couchbase -p couchbase -b beer-sample -m 100 -d ./samples/beer-sample.zip 2018-08-12T14:30:36.160-07:00 INFO ClusterExecutionITBase [main+] >> Errors occurred during the index creation phase. See logs for details. 2018-08-12T14:30:36.163-07:00 INFO TestExecutor [main] cbdocloader exit code 1 2018-08-12T14:30:40.075-07:00 INFO TestExecutor [pool-364-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-cc-single-node/kill-cc-single-node.003.adm 2018-08-12T14:30:40.076-07:00 INFO TestExecutor [pool-364-thread-1] Actual results file: target/rttest/results/recovery/kill-cc-single-node/kill-cc-single-node.003.adm 2018-08-12T14:30:40.076-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-cc-single-node/kill-cc-single-node.003.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 0 } 2018-08-12T14:30:41.791-07:00 INFO TestExecutor [pool-364-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-cc-single-node/kill-cc-single-node.003.adm 2018-08-12T14:30:41.792-07:00 INFO TestExecutor [pool-364-thread-1] Actual results file: target/rttest/results/recovery/kill-cc-single-node/kill-cc-single-node.003.adm 2018-08-12T14:30:41.792-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-cc-single-node/kill-cc-single-node.003.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 3238 } 2018-08-12T14:30:42.982-07:00 INFO TestExecutor [pool-364-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-cc-single-node/kill-cc-single-node.003.adm 2018-08-12T14:30:43.399-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/recovery/kill-cc-single-node/kill-cc-single-node.004.adm 2018-08-12T14:30:43.400-07:00 INFO ClusterExecutionITBase [main] Killing node 0 (signalNum: 9) 2018-08-12T14:30:43.743-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:30:43.807-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:30:55.176-07:00 INFO TestExecutor [pool-369-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-cc-single-node/kill-cc-single-node.008.adm 2018-08-12T14:30:55.177-07:00 INFO TestExecutor [pool-369-thread-1] Actual results file: target/rttest/results/recovery/kill-cc-single-node/kill-cc-single-node.008.adm 2018-08-12T14:30:55.177-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-cc-single-node/kill-cc-single-node.008.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 786 } 2018-08-12T14:30:56.687-07:00 INFO TestExecutor [pool-369-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-cc-single-node/kill-cc-single-node.008.adm 2018-08-12T14:30:56.840-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/recovery/kill-cc-single-node/kill-cc-single-node.009.adm 2018-08-12T14:30:58.425-07:00 INFO TestExecutor [main] [TEST]: recovery/kill-cc-single-node PASSED 2018-08-12T14:30:58.426-07:00 INFO TestExecutor [main] Cleaning up after test case recovery: kill-cc-single-node... 2018-08-12T14:30:58.516-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:30:58.585-07:00 INFO TestExecutor [main] Disconnecting link Metadata.Local 2018-08-12T14:30:59.142-07:00 INFO TestExecutor [main] Clean up completed for test case recovery: kill-cc-single-node 2018-08-12T14:30:59.142-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:30:59.145-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:30:59.805-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:30:59.813-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:30:59.813-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:31:01.823-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 18: recovery: kill-cc-single-node]'... 2018-08-12T14:31:01.970-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 18: recovery: kill-cc-single-node] SUCCEEDED 2018-08-12T14:31:01.971-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 19: recovery: kill-cc-multiple-nodes] START 2018-08-12T14:31:02.043-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:31:11.839-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:31:11.875-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:31:11.875-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:31:11.875-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:31:16.443-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:31:16.447-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:31:16.704-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:31:16.707-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:31:16.707-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-27-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:31:30.353-07:00 INFO TestExecutor [main] Starting [TEST]: recovery/kill-cc-multiple-nodes ... 2018-08-12T14:31:30.354-07:00 INFO CBDocLoaderCommand [main] Executing: sh -c ./bin/cbdocloader -c 127.0.0.1:9000 -u couchbase -p couchbase -b beer-sample -m 100 -d ./samples/beer-sample.zip 2018-08-12T14:31:39.705-07:00 INFO ClusterExecutionITBase [main+] >> Errors occurred during the index creation phase. See logs for details. 2018-08-12T14:31:39.708-07:00 INFO TestExecutor [main] cbdocloader exit code 1 2018-08-12T14:31:39.709-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1, 2] 2018-08-12T14:31:39.721-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1, 2]]) are fully removed 2018-08-12T14:31:39.729-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:31:39.730-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1, 2] with service(s): [ANALYTICS] 2018-08-12T14:31:44.350-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:31:44.625-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:31:44.629-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9002,127.0.0.1:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:31:52.927-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:31:52.932-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:32:07.330-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:32:07.330-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:32:07.334-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 3 2018-08-12T14:32:07.334-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-28-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:32:07.334-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-28-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:32:07.337-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-28-/127.0.0.1:9600] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:32:21.430-07:00 INFO ClusterExecutionITBase [main] Killing node 0 (signalNum: 9) 2018-08-12T14:32:23.468-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:32:23.701-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:32:28.704-07:00 INFO KvUpserter [Thread-400] KvUpserter stopped. Successful upserts: 2069, Failed upserts: 0 2018-08-12T14:32:28.734-07:00 ERRO asterix [pool-386-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:32:28.738-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:32:32.830-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:33.936-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:34.948-07:00 ERRO asterix [pool-389-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:32:34.951-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:32:36.027-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:37.430-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:38.816-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:39.897-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:40.936-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:42.164-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:43.230-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:44.284-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:45.347-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:46.426-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:47.487-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:48.590-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:49.638-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:50.683-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:51.743-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:52.787-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:53.824-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:54.909-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:55.945-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:56.983-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:58.029-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:32:59.071-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:00.144-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:01.190-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:02.229-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:03.289-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:04.328-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:05.383-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:06.437-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:07.489-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:08.549-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:09.593-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:10.672-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:11.725-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:12.760-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:13.798-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:14.847-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:15.900-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:16.947-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:17.986-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:19.032-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:20.081-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:21.120-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:22.160-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:23.193-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:24.234-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:25.275-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:26.323-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:27.365-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:28.422-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:33:29.434-07:00 ERRO TestExecutor [main] testFile src/test/resources/runtimets/queries/recovery/kill-cc-multiple-nodes/kill-cc-multiple-nodes.012.polldynamic.sqlpp raised an unexpected exception java.lang.Exception: Poll limit (60s) exceeded without obtaining expected result at org.apache.asterix.test.common.TestExecutor.poll(TestExecutor.java:1344) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at org.apache.asterix.test.common.TestExecutor.polldynamic(TestExecutor.java:1254) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at org.apache.asterix.test.common.TestExecutor.executeTestFile(TestExecutor.java:878) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at com.couchbase.analytics.test.CBASTestExecutor.syncDoExecuteTestFile(CBASTestExecutor.java:240) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.0.0-0000] at com.couchbase.analytics.test.CBASTestExecutor.doExecuteTestFile(CBASTestExecutor.java:210) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.0.0-0000] at com.couchbase.analytics.test.CBASTestExecutor.executeTestFile(CBASTestExecutor.java:192) ~[cbas-server-1.0.0-SNAPSHOT-tests.jar:6.0.0-0000] at org.apache.asterix.test.common.TestExecutor.executeTest(TestExecutor.java:1619) [asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at com.couchbase.analytics.test.cbserver.ClusterExecutionITBase.runTest(ClusterExecutionITBase.java:106) [test-classes/:?] at com.couchbase.analytics.test.cbserver.ClusterExecutionITJava10.test(ClusterExecutionITJava10.java:42) [test-classes/:?] at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_131] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.junit.runners.Suite.runChild(Suite.java:128) [junit-4.12.jar:4.12] at org.junit.runners.Suite.runChild(Suite.java:27) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) [surefire-booter-2.21.0.jar:2.21.0] 2018-08-12T14:33:29.438-07:00 ERRO asterix [pool-440-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 java.lang.IllegalStateException: Connection pool shut down at org.apache.http.util.Asserts.check(Asserts.java:34) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:185) ~[httpcore-4.4.5.jar:4.4.5] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:251) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:175) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] 2018-08-12T14:33:29.969-07:00 INFO TestExecutor [main] Diagnostic output: { "nodes" : [ { "config" : { "activeMemoryGlobalBudget" : 67108864, "activeStopTimeout" : 3600, "activeSuspendTimeout" : 3600, "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpAddress" : "127.0.0.1", "analyticsCcHttpPort" : 9601, "analyticsHttpAdminListenAddress" : "0.0.0.0", "analyticsHttpAdminListenPort" : 9641, "analyticsHttpAdminPublicAddress" : "127.0.0.1", "analyticsHttpAdminPublicPort" : 9641, "analyticsHttpAdminRequestQueueSize" : 32, "analyticsHttpAdminThreadCount" : 4, "analyticsHttpListenAddress" : "0.0.0.0", "analyticsHttpListenPort" : 9630, "analyticsHttpPublicAddress" : "127.0.0.1", "analyticsHttpPublicPort" : 9630, "analyticsHttpRequestQueueSize" : 256, "analyticsHttpThreadCount" : 16, "analyticsHttpsListenPort" : 19602, "analyticsNodeName" : "127.0.0.1:9002", "analyticsSslEnabled" : true, "bucketConfigAttemptTimeout" : 15, "bucketConfigTotalTimeout" : 60, "cbasAddress" : "127.0.0.1", "cbasPort" : 9640, "clusterAddress" : "127.0.0.1", "clusterConnectRetries" : 5, "clusterListenAddress" : "0.0.0.0", "clusterListenPort" : 9635, "clusterPort" : 9602, "clusterPublicAddress" : "127.0.0.1", "clusterPublicPort" : 9635, "collectDcpStateFromNodesTimeout" : 60, "compilerFramesize" : 32768, "compilerGroupmemory" : 33554432, "compilerJoinmemory" : 33554432, "compilerParallelism" : 0, "compilerSortmemory" : 33554432, "compilerStringoffset" : 0, "compilerTextsearchmemory" : 33554432, "coreDumpDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0/coredump", "dataListenAddress" : "0.0.0.0", "dataListenPort" : 9636, "dataPublicAddress" : "127.0.0.1", "dataPublicPort" : 9636, "dcpChannelConnectAttemptTimeout" : 15, "dcpChannelConnectTotalTimeout" : 60, "deadlockWatchdogHaltDelaySeconds" : 120, "deadlockWatchdogPollSeconds" : 300, "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "driverArgs" : [ "--jsonconfig", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0/config.json", "--nc", "-node-id", "200758d4b55bbd40c0c91e8c2cc11473" ], "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_2" ], "jvmArgs" : null, "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_2", "logLevel" : "DEBUG", "maxWaitActiveCluster" : 60, "maxWebRequestSize" : 52428800, "messagingFrameCount" : 512, "messagingFrameSize" : 4096, "messagingListenAddress" : "0.0.0.0", "messagingListenPort" : 9638, "messagingPublicAddress" : "127.0.0.1", "messagingPublicPort" : 9638, "metadataCallbackPort" : 9644, "metadataListenPort" : 9643, "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "metadataRegistrationTimeoutSecs" : 60, "ncserviceAddress" : "127.0.0.1", "ncservicePid" : -1, "ncservicePort" : -1, "netBufferCount" : 1, "netThreadCount" : 1, "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9002, "publicAddress" : "127.0.0.1", "replicationListenAddress" : "0.0.0.0", "replicationListenPort" : 9642, "replicationPublicAddress" : "127.0.0.1", "replicationPublicPort" : 9642, "resultListenAddress" : "0.0.0.0", "resultListenPort" : 9637, "resultManagerMemory" : -1, "resultPublicAddress" : "127.0.0.1", "resultPublicPort" : 9637, "resultSweepThreshold" : 60000, "resultTtl" : 86400000, "startingPartitionId" : 6, "storageBuffercacheMaxopenfiles" : 2147483647, "storageBuffercachePagesize" : 131072, "storageBuffercacheSize" : 805306368, "storageLsmBloomfilterFalsepositiverate" : 0.01, "storageMaxActiveWritableDatasets" : 8, "storageMemorycomponentGlobalbudget" : 805306368, "storageMemorycomponentNumcomponents" : 2, "storageMemorycomponentPagesize" : 131072, "storageMetadataMemorycomponentNumpages" : 8, "threaddumpFrequencySeconds" : 300, "threaddumpLogLevel" : "DEBUG", "traceCategories" : [ ], "txnCommitprofilerEnabled" : false, "txnCommitprofilerReportinterval" : 5, "txnJobRecoveryMemorysize" : 67108864, "txnLockEscalationthreshold" : 1000, "txnLockShrinktimer" : 5000, "txnLockTimeoutSweepthreshold" : 10000, "txnLockTimeoutWaitthreshold" : 60000, "txnLogBufferNumpages" : 8, "txnLogBufferPagesize" : 262144, "txnLogCheckpointHistory" : 0, "txnLogCheckpointLsnthreshold" : 67108864, "txnLogCheckpointPollfrequency" : 120, "txnLogDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0/txn-log", "txnLogPartitionsize" : 268435456 }, "date" : "Sun Aug 12 14:33:29 PDT 2018", "metakv" : { "bootstrap/ensureCc/e8a78c35fcb3323712f05201c598bcde" : "MjAxOC0wOC0xMlQyMTozMjozMi4yOTE1MDda", "cluster/state/metadata/replica" : { "nodes" : [ { "location" : "127.0.0.1:9627", "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac" }, { "location" : "127.0.0.1:9642", "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473" } ], "revision" : 2 }, "config/node/200758d4b55bbd40c0c91e8c2cc11473" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9641", "analyticsHttpListenPort" : "9630", "analyticsHttpsListenPort" : "19602", "analyticsNodeName" : "127.0.0.1:9002", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9640", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9631", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9633", "ccClusterListenPort" : "9632", "ccConsoleListenPort" : "9634", "ccControllerId" : "2", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9635", "clusterPort" : "9602", "dataListenPort" : "9636", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_2", "messagingListenPort" : "9638", "metadataCallbackPort" : "9644", "metadataListenPort" : "9643", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9002, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9642, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9637", "startingPartitionId" : "6", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473" }, "config/node/d7a317d45ea35c6779cbb2ba8d1f7eac" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9626", "analyticsHttpListenPort" : "9615", "analyticsHttpsListenPort" : "19601", "analyticsNodeName" : "127.0.0.1:9001", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9625", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9616", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9618", "ccClusterListenPort" : "9617", "ccConsoleListenPort" : "9619", "ccControllerId" : "1", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9620", "clusterPort" : "9602", "dataListenPort" : "9621", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_1", "messagingListenPort" : "9623", "metadataCallbackPort" : "9629", "metadataListenPort" : "9628", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9001, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9627, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9622", "startingPartitionId" : "3", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac" }, "config/node/e8a78c35fcb3323712f05201c598bcde" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9611", "analyticsHttpListenPort" : "9600", "analyticsHttpsListenPort" : "19600", "analyticsNodeName" : "127.0.0.1:9000", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9610", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9601", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9603", "ccClusterListenPort" : "9602", "ccConsoleListenPort" : "9604", "ccControllerId" : "0", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9605", "clusterPort" : "9602", "dataListenPort" : "9606", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_0", "messagingListenPort" : "9608", "metadataCallbackPort" : "9614", "metadataListenPort" : "9613", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9000, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9612, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9607", "startingPartitionId" : "0", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "e8a78c35fcb3323712f05201c598bcde" }, "nextControllerId" : 3, "nextPartitionId" : 9, "topology" : { "balanceState" : "balanced", "ccNodeId" : "e8a78c35fcb3323712f05201c598bcde", "configVersion" : 1, "id" : "147bdd4381a8eb541c73daabddf1d333", "keepNodesUpdated" : false, "metadataNodeId" : "e8a78c35fcb3323712f05201c598bcde", "metadataPartition" : 0, "nodes" : [ { "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "opaque" : { "cc-http-port" : "9601", "controller-id" : "0", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "0" }, "priority" : 0 }, { "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "opaque" : { "cc-http-port" : "9616", "controller-id" : "1", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "3" }, "priority" : 0 }, { "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "opaque" : { "cc-http-port" : "9631", "controller-id" : "2", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "6" }, "priority" : 0 } ], "rev" : 2, "type" : "topology-change-rebalance" } }, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "nodeName" : "127.0.0.1:9002", "runtime" : { "bootDate" : "Sun Aug 12 14:31:54 PDT 2018", "daysWithoutAnAccident" : "0.00", "inputArguments" : [ "-XX:+ExitOnOutOfMemoryError", "-XX:MaxGCPauseMillis=60000", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9639", "-Xmx3072m", "-Dfile.encoding=UTF-8", "-Djava.net.preferIPv4Stack=true", "-Djava.net.preferIPv6Addresses=false", "--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED", "--add-opens=java.management/sun.management=ALL-UNNAMED", "-DlogDir=/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_2" ], "numProcessors" : 4, "pid" : 21445, "systemProperties" : { "awt.toolkit" : "sun.awt.X11.XToolkit", "com.couchbase.allowReverseDns" : "false", "com.couchbase.forceIPv4" : "false", "file.encoding" : "UTF-8", "file.encoding.pkg" : "sun.io", "file.separator" : "/", "java.awt.graphicsenv" : "sun.awt.X11GraphicsEnvironment", "java.awt.printerjob" : "sun.print.PSPrinterJob", "java.class.path" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-compiler.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-core.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-data.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-rewriter.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-runtime.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/args4j-2.33.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-active.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-algebra.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-app.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-external-data.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-aql.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-sqlpp.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-metadata.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-om.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-replication.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-runtime.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-transactions.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-connector.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-dcp-client.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-server.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-codec-1.9.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-collections4-4.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-compress-1.12.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-io-2.6.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-lang3-3.7.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-logging-1.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-text-1.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/core-io-1.5.6.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/disruptor-1.2.13-jdk8.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/esri-geometry-api-2.0.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/fastutil-8.1.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/guava-18.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/httpclient-4.5.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/httpcore-4.4.5.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-api.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-client.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-comm.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-cc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-nc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-data-std.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-dataflow-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-dataflow-std.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-http.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-ipc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-net.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-bloomfilter.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-btree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-btree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-invertedindex.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-rtree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-rtree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-util.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/ini4j-0.5.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-annotations-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-core-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-databind-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-api-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-core-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-jcl-2.11.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-jul-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-slf4j-impl-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-web-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/netty-all-4.1.25.Final.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/netty-tcnative-boringssl-static-2.0.8.Final.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/rxjava-1.3.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/slf4j-api-1.8.0-alpha2.jar" ], "java.class.version" : "54.0", "java.home" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/.jre/jre-10.0.2", "java.io.tmpdir" : "/tmp", "java.library.path" : [ "/usr/java/packages/lib", "/usr/lib64", "/lib64", "/lib", "/usr/lib" ], "java.net.preferIPv4Stack" : "true", "java.net.preferIPv6Addresses" : "false", "java.rmi.server.hostname" : "127.0.0.1", "java.runtime.name" : "Java(TM) SE Runtime Environment", "java.runtime.version" : "10.0.2+13", "java.specification.name" : "Java Platform API Specification", "java.specification.vendor" : "Oracle Corporation", "java.specification.version" : "10", "java.util.logging.manager" : "org.apache.logging.log4j.jul.LogManager", "java.vendor" : "Oracle Corporation", "java.vendor.url" : "http://java.oracle.com/", "java.vendor.url.bug" : "http://bugreport.java.com/bugreport/", "java.vendor.version" : "18.3", "java.version" : "10.0.2", "java.version.date" : "2018-07-17", "java.vm.compressedOopsMode" : "Zero based", "java.vm.info" : "mixed mode", "java.vm.name" : "Java HotSpot(TM) 64-Bit Server VM", "java.vm.specification.name" : "Java Virtual Machine Specification", "java.vm.specification.vendor" : "Oracle Corporation", "java.vm.specification.version" : "10", "java.vm.vendor" : "\"Oracle Corporation\"", "java.vm.version" : "10.0.2+13", "jdk.debug" : "release", "line.separator" : "\n", "log4j2.shutdownHookEnabled" : "false", "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_2", "os.arch" : "amd64", "os.name" : "Linux", "os.version" : "4.4.0-104-generic", "path.separator" : ":", "sun.arch.data.model" : "64", "sun.boot.library.path" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/.jre/jre-10.0.2/lib" ], "sun.cpu.endian" : "little", "sun.cpu.isalist" : "", "sun.io.unicode.encoding" : "UnicodeLittle", "sun.java.command" : [ "com.couchbase.analytics.control.AnalyticsDriver", "--jsonconfig", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0/config.json", "--nc", "-node-id", "200758d4b55bbd40c0c91e8c2cc11473" ], "sun.java.launcher" : "SUN_STANDARD", "sun.jnu.encoding" : "UTF-8", "sun.management.compiler" : "HotSpot 64-Bit Tiered Compilers", "sun.nio.ch.bugLevel" : "", "sun.os.patch.level" : "unknown", "user.country" : "US", "user.dir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server", "user.home" : "/home/couchbase", "user.language" : "en", "user.name" : "couchbase", "user.timezone" : "America/Los_Angeles" }, "uptimeMillis" : 95480 }, "stats" : { "applicationCpuCoreBudget" : 4, "applicationMemoryBudget" : 1610612736, "date" : 1534109609525, "diskRead" : 0, "diskWrite" : 11317248, "gcs" : [ { "collection_count" : null, "collection_time" : null, "name" : "G1 Young Generation" }, { "collection_count" : null, "collection_time" : null, "name" : "G1 Old Generation" } ], "heapCommittedSize" : 264241152, "heapInitSize" : 264241152, "heapMaxSize" : 3221225472, "heapUsedSize" : 103463936, "heartbeatTime" : 1534109582074, "ipcMessageBytesReceived" : 168959, "ipcMessageBytesSent" : 198464, "ipcMessagesReceived" : 30, "ipcMessagesSent" : 36, "netPayloadBytesRead" : 16613916, "netPayloadBytesWritten" : 6619676, "netSignalingBytesRead" : 6848, "netSignalingBytesWritten" : 6872, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "nonheapCommittedSize" : 78667776, "nonheapInitSize" : 7667712, "nonheapMaxSize" : -1, "nonheapUsedSize" : 75390704, "peakThreadCount" : 105, "resultNetPayloadBytesRead" : 0, "resultNetPayloadBytesWritten" : 0, "resultNetSignalingBytesRead" : 0, "resultNetSignalingBytesWritten" : 0, "systemLoadAverage" : 6.49, "threadCount" : 105 }, "threaddump" : { "date" : "Sun Aug 12 14:33:29 PDT 2018", "deadlockedThreadIds" : [ 28, 68 ], "monitorDeadlockedThreadIds" : [ 28, 68 ], "threads" : [ { "blocked_count" : 11, "id" : 1, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@43e53f6d", "name" : "main", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquire(Unknown Source)", "app//com.couchbase.analytics.control.AnalyticsDriver.main(AnalyticsDriver.java:111)" ], "state" : "WAITING" }, { "blocked_count" : 1, "id" : 2, "name" : "Reference Handler", "stack" : [ "java.base@10.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method)", "java.base@10.0.2/java.lang.ref.Reference.processPendingReferences(Unknown Source)", "java.base@10.0.2/java.lang.ref.Reference.access$000(Unknown Source)", "java.base@10.0.2/java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 4, "id" : 3, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@14ccb84f", "name" : "Finalizer", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 4, "name" : "Signal Dispatcher", "stack" : [ ], "state" : "RUNNABLE" }, { "blocked_count" : 3, "id" : 9, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@410b8885", "name" : "Common-Cleaner", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/jdk.internal.ref.CleanerImpl.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)", "java.base@10.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 10, "name" : "JDWP Transport Listener: dt_socket", "stack" : [ ], "state" : "RUNNABLE" }, { "id" : 11, "name" : "JDWP Event Helper Thread", "stack" : [ ], "state" : "RUNNABLE" }, { "id" : 14, "name" : "AsyncAppender-AsyncDebugLog", "stack" : [ "java.base@10.0.2/java.lang.Thread.yield(Native Method)", "app//com.conversantmedia.util.concurrent.Condition.progressiveYield(Condition.java:68)", "app//com.conversantmedia.util.concurrent.AbstractWaitingCondition.await(AbstractWaitingCondition.java:144)", "app//com.conversantmedia.util.concurrent.DisruptorBlockingQueue.take(DisruptorBlockingQueue.java:245)", "app//org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:403)" ], "state" : "RUNNABLE" }, { "id" : 16, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@54e85737", "name" : "ShutdownWatchdog", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquireUninterruptibly(Unknown Source)", "app//org.apache.hyracks.util.ExitUtil$ShutdownWatchdog.run(ExitUtil.java:109)" ], "state" : "WAITING" }, { "id" : 18, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e1c0490", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 501951555 } ], "name" : "IoRequestHandler-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 19, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e1c0490", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1002041276 } ], "name" : "IoRequestHandler-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 20, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e1c0490", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 485475507 } ], "name" : "IoRequestHandler-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 21, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e1c0490", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 724028528 } ], "name" : "IoRequestHandler-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 22, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e1c0490", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1869247587 } ], "name" : "IoRequestHandler-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 23, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3e1c0490", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 467313213 } ], "name" : "IoRequestHandler-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 2, "id" : 25, "lock_name" : "java.util.TaskQueue@4f63cddc", "name" : "Timer-0", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.util.TimerThread.mainLoop(Unknown Source)", "java.base@10.0.2/java.util.TimerThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 26, "locked_monitors" : [ { "className" : "java.io.BufferedInputStream", "identityHashCode" : 1387342613, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "java.io.BufferedInputStream", "fileName" : null, "lineNumber" : -1, "methodName" : "read", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "Stdin Watcher", "stack" : [ "java.base@10.0.2/java.io.FileInputStream.readBytes(Native Method)", "java.base@10.0.2/java.io.FileInputStream.read(Unknown Source)", "java.base@10.0.2/java.io.BufferedInputStream.fill(Unknown Source)", "java.base@10.0.2/java.io.BufferedInputStream.read(Unknown Source)", "app//com.couchbase.analytics.control.StdInWatcher.lambda$start$0(StdInWatcher.java:38)", "app//com.couchbase.analytics.control.StdInWatcher$$Lambda$140/1802736936.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 19, "id" : 27, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1148620699, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 445659905, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1271662754, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:9635]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "blocked_count" : 51, "id" : 28, "lock_name" : "org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock@6084348e", "lock_owner_id" : 68, "locked_monitors" : [ { "className" : "org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection", "identityHashCode" : 276724557, "lockedStackDepth" : 4, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "org.apache.hyracks.net.protocols.muxdemux.ChannelSet", "fileName" : "ChannelSet.java", "lineNumber" : 203, "methodName" : "notifyIOError", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } }, { "className" : "org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection", "identityHashCode" : 276724557, "lockedStackDepth" : 5, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection", "fileName" : "MultiplexedConnection.java", "lineNumber" : 123, "methodName" : "notifyIOError", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9636]", "stack" : [ "app//org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface$CloseableBufferAcceptor.error(AbstractChannelWriteInterface.java:150)", "app//org.apache.hyracks.comm.channels.NetworkOutputChannel.abort(NetworkOutputChannel.java:108)", "app//org.apache.hyracks.control.nc.net.NetworkManager$InitialBufferAcceptor.error(NetworkManager.java:142)", "app//org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock.reportRemoteError(ChannelControlBlock.java:127)", "app//org.apache.hyracks.net.protocols.muxdemux.ChannelSet.notifyIOError(ChannelSet.java:203)", "app//org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection.notifyIOError(MultiplexedConnection.java:123)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:189)" ], "state" : "BLOCKED" }, { "id" : 29, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7bd743ac", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2081952435 } ], "name" : "Executor-1:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.transaction.management.service.logging.LogManager$FlushLogsLogger.run(LogManager.java:644)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 12, "id" : 30, "lock_name" : "org.apache.asterix.transaction.management.service.logging.LogBuffer@50f5dd6c", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 13001549 } ], "name" : "Executor-2:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.Object.wait(Unknown Source)", "app//org.apache.asterix.transaction.management.service.logging.LogBuffer.flush(LogBuffer.java:176)", "app//org.apache.asterix.transaction.management.service.logging.LogFlusher.call(LogManager.java:699)", "app//org.apache.asterix.transaction.management.service.logging.LogFlusher.call(LogManager.java:654)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 31, "lock_name" : "org.apache.asterix.replication.logging.ReplicationLogBuffer@3332702c", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1591683105 } ], "name" : "TxnLog Replicator", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.Object.wait(Unknown Source)", "app//org.apache.asterix.replication.logging.ReplicationLogBuffer.flush(ReplicationLogBuffer.java:94)", "app//org.apache.asterix.replication.logging.TxnLogReplicator.call(TxnLogReplicator.java:93)", "app//org.apache.asterix.replication.logging.TxnLogReplicator.call(TxnLogReplicator.java:31)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 32, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@72530280", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1090057640 } ], "name" : "ReplicationJobsProcessor", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.replication.management.IndexReplicationManager$ReplicationJobsProcessor.run(IndexReplicationManager.java:174)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 34, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3358f545", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1779219567 } ], "name" : "200758d4b55bbd40c0c91e8c2cc11473RemoteLogsNotifier", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.replication.logging.RemoteLogsNotifier.run(RemoteLogsNotifier.java:63)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 35, "lock_name" : "java.lang.Object@4d3850ff", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1501716646 } ], "name" : "Executor-6:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "app//org.apache.hyracks.storage.common.buffercache.DelayPageCleanerPolicy.notifyCleanCycleFinish(DelayPageCleanerPolicy.java:34)", "app//org.apache.hyracks.storage.common.buffercache.BufferCache$CleanerThread.runCleanCycle(BufferCache.java:773)", "app//org.apache.hyracks.storage.common.buffercache.BufferCache$CleanerThread.run(BufferCache.java:745)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 36, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6dabfa64", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 890733699 } ], "name" : "Executor-7:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.messaging.NCMessageBroker$MessageDeliveryService.run(NCMessageBroker.java:162)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 37, "lock_name" : "com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor@786b253", "name" : "Thread-3", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "app//com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor.run(NsServerHttpClient.java:66)" ], "state" : "TIMED_WAITING" }, { "id" : 38, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 801996095 } ], "name" : "Executor-8:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//org.apache.hyracks.control.common.result.ResultStateSweeper.run(ResultStateSweeper.java:44)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 39, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1937808699, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2091947036, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1325021492, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9637]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "id" : 40, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 2094796434, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 188886391, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1940543607, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9638]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "blocked_count" : 1, "id" : 41, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@6c06fcaf", "name" : "RMI RenewClean-[127.0.0.1:9614]", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 42, "lock_name" : "sun.rmi.transport.GC$LatencyLock@7032d723", "name" : "RMI GC Daemon", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.rmi@10.0.2/sun.rmi.transport.GC$Daemon.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)", "java.base@10.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 43, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a8165c7", "name" : "RMI Scheduler(0)", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 3, "id" : 44, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@4bc9f100", "name" : "200758d4b55bbd40c0c91e8c2cc11473-Heartbeat", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.tryAcquire(Unknown Source)", "app//org.apache.hyracks.control.nc.heartbeat.HeartbeatTask.run(HeartbeatTask.java:59)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 2, "id" : 24, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10f18f52", "name" : "Worker:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:114)" ], "state" : "WAITING" }, { "blocked_count" : 1, "id" : 45, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@769cc9a0", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1284159574 } ], "name" : "org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.2:TAID:TID:ANID:ODID:0:0:8:0:0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquireUninterruptibly(Unknown Source)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.cancelTasks(SuperActivityOperatorNodePushable.java:245)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:232)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:88)", "app//org.apache.hyracks.control.nc.Task.run(Task.java:297)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 46, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 399648233, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 320313314, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 251662797, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:45576]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "id" : 47, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fc10844", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1675603360 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileRequestHandler", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileRequestHandler.run(HyracksConnection.java:431)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 48, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2105078741 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileHandlerWatcher", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "java.base@10.0.2/java.lang.Thread.sleep(Unknown Source)", "java.base@10.0.2/java.util.concurrent.TimeUnit.sleep(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileHandlerWatcher.run(HyracksConnection.java:459)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 50, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@19ba9b38", "name" : "ObjectCleanerThread", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "app//io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 51, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 910010218, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 968640025, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 774437442, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 33, "locked_monitors" : [ { "className" : "java.lang.Object", "identityHashCode" : 1661392868, "lockedStackDepth" : 2, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.ServerSocketChannelImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "accept", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "200758d4b55bbd40c0c91e8c2cc11473 Replication Channel Thread", "stack" : [ "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)", "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)", "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)", "app//org.apache.asterix.replication.management.ReplicationChannel.run(ReplicationChannel.java:72)" ], "state" : "RUNNABLE" }, { "id" : 52, "name" : "Checkpoint Thread (52)", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//org.apache.asterix.transaction.management.service.recovery.CheckpointThread.run(CheckpointThread.java:61)" ], "state" : "TIMED_WAITING" }, { "id" : 54, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 945502745, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1670131589, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 178201436, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 10, "id" : 55, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1609788444, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1041829385, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1013706312, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-2", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 56, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 58, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 3, "id" : 59, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 197830953, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2111149855, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 514801475, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-3", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 60, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 61, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1021981875, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1656177126, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 864767936, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-4", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 62, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 63, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 184, "id" : 64, "locked_monitors" : [ { "className" : "org.apache.hyracks.control.common.NodeControllerData", "identityHashCode" : 1413037013, "lockedStackDepth" : 6, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "org.apache.hyracks.control.common.NodeControllerData", "fileName" : "NodeControllerData.java", "lineNumber" : 314, "methodName" : "toDetailedJSON", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } } ], "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1754757820 } ], "name" : "Executor-12:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/java.util.Spliterators.spliterator(Unknown Source)", "java.base@10.0.2/java.util.Arrays.spliterator(Unknown Source)", "java.base@10.0.2/java.util.Arrays.stream(Unknown Source)", "java.base@10.0.2/java.util.Arrays.stream(Unknown Source)", "java.base@10.0.2/java.util.stream.LongStream.of(Unknown Source)", "app//org.apache.hyracks.util.JSONUtil.put(JSONUtil.java:253)", "app//org.apache.hyracks.control.common.NodeControllerData.toDetailedJSON(NodeControllerData.java:314)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.processNodeStats(NodeDiagnosticsServlet.java:129)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$getResultJson$2(NodeDiagnosticsServlet.java:89)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$342/1354705878.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 65, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@399bc24d", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 491765511 } ], "name" : "org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.2:TAID:TID:ANID:ODID:0:0:6:0:0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquireUninterruptibly(Unknown Source)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.cancelTasks(SuperActivityOperatorNodePushable.java:245)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:232)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:88)", "app//org.apache.hyracks.control.nc.Task.run(Task.java:297)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 66, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@61fb0ce9", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2145170330 } ], "name" : "org.apache.hyracks.api.rewriter.runtime.SuperActivity:JID:0.2:TAID:TID:ANID:ODID:0:0:7:0:0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquireUninterruptibly(Unknown Source)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.cancelTasks(SuperActivityOperatorNodePushable.java:245)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:232)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:88)", "app//org.apache.hyracks.control.nc.Task.run(Task.java:297)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 217, "id" : 67, "lock_name" : "org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection@107e7b4d", "lock_owner_id" : 28, "locked_monitors" : [ { "className" : "org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock", "identityHashCode" : 141011545, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface$CloseableBufferAcceptor", "fileName" : "AbstractChannelWriteInterface.java", "lineNumber" : 151, "methodName" : "error", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } } ], "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 77107519 } ], "name" : "Executor-15:200758d4b55bbd40c0c91e8c2cc11473:JID:0.2:TAID:TID:ANID:ODID:0:0:6:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.beer-sample(CouchbaseMetadataExtension))[6]:BucketOperatorDescriptor", "stack" : [ "app//org.apache.hyracks.net.protocols.muxdemux.ChannelSet.markPendingWrite(ChannelSet.java:175)", "app//org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock.markPendingWrite(ChannelControlBlock.java:157)", "app//org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface.adjustChannelWritability(AbstractChannelWriteInterface.java:92)", "app//org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface$CloseableBufferAcceptor.error(AbstractChannelWriteInterface.java:151)", "app//org.apache.hyracks.comm.channels.NetworkOutputChannel.fail(NetworkOutputChannel.java:99)", "app//org.apache.hyracks.control.nc.partitions.PipelinedPartition.fail(PipelinedPartition.java:105)", "app//com.couchbase.analytics.runtime.ProgressPartitionDataWriter.fail(ProgressPartitionDataWriter.java:207)", "app//org.apache.hyracks.dataflow.std.base.AbstractReplicateOperatorDescriptor$ReplicatorMaterializerActivityNode$1.fail(AbstractReplicateOperatorDescriptor.java:187)", "app//org.apache.hyracks.api.util.CleanupUtils.fail(CleanupUtils.java:91)", "app//com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:54)", "app//org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:103)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$285/1360234498.run(Unknown Source)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:212)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$286/1654431569.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "BLOCKED" }, { "blocked_count" : 189, "id" : 68, "lock_name" : "org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection@107e7b4d", "lock_owner_id" : 28, "locked_monitors" : [ { "className" : "org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock", "identityHashCode" : 1619276942, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface$CloseableBufferAcceptor", "fileName" : "AbstractChannelWriteInterface.java", "lineNumber" : 127, "methodName" : "accept", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } }, { "className" : "java.lang.Object", "identityHashCode" : 362663699, "lockedStackDepth" : 17, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "com.couchbase.analytics.adapter.CouchbaseConnector", "fileName" : "CouchbaseConnector.java", "lineNumber" : 426, "methodName" : "next", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } } ], "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 432422469 } ], "name" : "Executor-16:200758d4b55bbd40c0c91e8c2cc11473:JID:0.2:TAID:TID:ANID:ODID:0:0:8:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.beer-sample(CouchbaseMetadataExtension))[8]:BucketOperatorDescriptor", "stack" : [ "app//org.apache.hyracks.net.protocols.muxdemux.ChannelSet.markPendingWrite(ChannelSet.java:175)", "app//org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock.markPendingWrite(ChannelControlBlock.java:157)", "app//org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface.adjustChannelWritability(AbstractChannelWriteInterface.java:92)", "app//org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface$CloseableBufferAcceptor.accept(AbstractChannelWriteInterface.java:127)", "app//org.apache.hyracks.comm.channels.NetworkOutputChannel.nextFrame(NetworkOutputChannel.java:93)", "app//org.apache.hyracks.control.nc.partitions.PipelinedPartition.nextFrame(PipelinedPartition.java:82)", "app//com.couchbase.analytics.runtime.ProgressFrameTupleAppender.forward(ProgressFrameTupleAppender.java:93)", "app//com.couchbase.analytics.runtime.ProgressFrameTupleAppender.write(ProgressFrameTupleAppender.java:73)", "app//com.couchbase.analytics.runtime.ProgressFrameTupleAppender.flush(ProgressFrameTupleAppender.java:105)", "app//org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.flush(AbstractFrameAppender.java:124)", "app//com.couchbase.analytics.runtime.ProgressPartitionDataWriter.flush(ProgressPartitionDataWriter.java:175)", "app//org.apache.hyracks.dataflow.std.base.AbstractReplicateOperatorDescriptor$ReplicatorMaterializerActivityNode$1.flush(AbstractReplicateOperatorDescriptor.java:149)", "app//org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.flush(AbstractFrameAppender.java:117)", "app//org.apache.asterix.external.dataflow.TupleForwarder.flush(TupleForwarder.java:47)", "app//org.apache.asterix.external.dataflow.AbstractFeedDataFlowController.flush(AbstractFeedDataFlowController.java:48)", "app//com.couchbase.analytics.adapter.CouchbaseConnector.pollNextMessage(CouchbaseConnector.java:453)", "app//com.couchbase.analytics.adapter.CouchbaseConnector.take(CouchbaseConnector.java:438)", "app//com.couchbase.analytics.adapter.CouchbaseConnector.next(CouchbaseConnector.java:426)", "app//org.apache.asterix.external.dataflow.FeedRecordDataFlowController.next(FeedRecordDataFlowController.java:134)", "app//org.apache.asterix.external.dataflow.FeedRecordDataFlowController.start(FeedRecordDataFlowController.java:82)", "app//org.apache.asterix.external.dataset.adapter.FeedAdapter.start(FeedAdapter.java:38)", "app//com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:52)", "app//org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:103)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$285/1360234498.run(Unknown Source)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:212)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$286/1654431569.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "BLOCKED" }, { "blocked_count" : 235, "id" : 69, "lock_name" : "org.apache.hyracks.net.protocols.muxdemux.MultiplexedConnection@107e7b4d", "lock_owner_id" : 28, "locked_monitors" : [ { "className" : "org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock", "identityHashCode" : 1928592967, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface$CloseableBufferAcceptor", "fileName" : "AbstractChannelWriteInterface.java", "lineNumber" : 151, "methodName" : "error", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } } ], "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2111985477 } ], "name" : "Executor-17:200758d4b55bbd40c0c91e8c2cc11473:JID:0.2:TAID:TID:ANID:ODID:0:0:7:0:SuperActivityOperatorNodePushable:BucketOperatorNodePushable:(Default.Local.beer-sample(CouchbaseMetadataExtension))[7]:BucketOperatorDescriptor", "stack" : [ "app//org.apache.hyracks.net.protocols.muxdemux.ChannelSet.markPendingWrite(ChannelSet.java:175)", "app//org.apache.hyracks.net.protocols.muxdemux.ChannelControlBlock.markPendingWrite(ChannelControlBlock.java:157)", "app//org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface.adjustChannelWritability(AbstractChannelWriteInterface.java:92)", "app//org.apache.hyracks.net.protocols.muxdemux.AbstractChannelWriteInterface$CloseableBufferAcceptor.error(AbstractChannelWriteInterface.java:151)", "app//org.apache.hyracks.comm.channels.NetworkOutputChannel.fail(NetworkOutputChannel.java:99)", "app//org.apache.hyracks.control.nc.partitions.PipelinedPartition.fail(PipelinedPartition.java:105)", "app//com.couchbase.analytics.runtime.ProgressPartitionDataWriter.fail(ProgressPartitionDataWriter.java:207)", "app//org.apache.hyracks.dataflow.std.base.AbstractReplicateOperatorDescriptor$ReplicatorMaterializerActivityNode$1.fail(AbstractReplicateOperatorDescriptor.java:187)", "app//org.apache.hyracks.api.util.CleanupUtils.fail(CleanupUtils.java:91)", "app//com.couchbase.analytics.runtime.BucketOperatorNodePushable.start(BucketOperatorNodePushable.java:54)", "app//org.apache.asterix.active.ActiveSourceOperatorNodePushable.initialize(ActiveSourceOperatorNodePushable.java:103)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$285/1360234498.run(Unknown Source)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:212)", "app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$286/1654431569.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "BLOCKED" }, { "blocked_count" : 5, "id" : 70, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1263937716, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2127984357, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 88660859, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-5", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 71, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 72, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 73, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 419445719 } ], "name" : "HttpExecutor(port:9641)-5", "stack" : [ "java.base@10.0.2/jdk.internal.org.objectweb.asm.FieldWriter.put(Unknown Source)", "java.base@10.0.2/jdk.internal.org.objectweb.asm.ClassWriter.toByteArray(Unknown Source)", "java.base@10.0.2/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(Unknown Source)", "java.base@10.0.2/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(Unknown Source)", "java.base@10.0.2/java.lang.invoke.LambdaMetafactory.metafactory(Unknown Source)", "java.base@10.0.2/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(Unknown Source)", "java.base@10.0.2/java.lang.invoke.LambdaForm$MH/1009326765.invoke(LambdaForm$MH)", "java.base@10.0.2/java.lang.invoke.Invokers$Holder.invoke_MT(Unknown Source)", "java.base@10.0.2/java.lang.invoke.CallSite.makeSite(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandleNatives.linkCallSite(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.getTimeZoneNameProvider(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.getLocaleServiceProvider(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.LocaleServiceProviderPool.findProviders(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayNamesImpl(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayName(Unknown Source)", "java.base@10.0.2/java.util.TimeZone.getDisplayName(Unknown Source)", "java.base@10.0.2/java.util.Date.toString(Unknown Source)", "java.base@10.0.2/java.lang.String.valueOf(Unknown Source)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.getResultJson(NodeDiagnosticsServlet.java:100)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.get(NodeDiagnosticsServlet.java:74)", "app//org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:88)", "app//com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:79)", "app//org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:71)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:56)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 74, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-6", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 75, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-7", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 76, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-8", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 77, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-9", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 78, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-10", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 193, "id" : 83, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@124d9a7b", "name" : "Executor-18:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 85, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@124d9a7b", "name" : "Executor-20:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 210, "id" : 84, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1572860155 } ], "name" : "Executor-19:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.management@10.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)", "java.management@10.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)", "java.management@10.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)", "app//org.apache.hyracks.util.ThreadDumpUtil.takeDumpJSON(ThreadDumpUtil.java:57)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$getResultJson$0(NodeDiagnosticsServlet.java:86)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$340/293741037.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 1, "id" : 86, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@124d9a7b", "name" : "Executor-21:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 87, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 58362838 } ], "name" : "Executor-22:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "java.base@10.0.2/java.lang.Thread.sleep(Unknown Source)", "java.base@10.0.2/java.util.concurrent.TimeUnit.sleep(Unknown Source)", "app//org.apache.hyracks.control.nc.work.EnsureAllCcTasksCompleted.run(EnsureAllCcTasksCompleted.java:60)", "java.base@10.0.2/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 88, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-11", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 46, "id" : 80, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 1600239497, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 472988701, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1421831516, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-2", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 45, "id" : 81, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 719305714, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2026933471, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1838450188, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-3", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 84, "id" : 79, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 1122943855, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 364503275, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1777541367, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 89, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-12", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 90, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-13", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 91, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-14", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 93, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201e7f23", "name" : "FixerThread:cbas:beer-sample:200758d4b55bbd40c0c91e8c2cc11473:8", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.poll(Unknown Source)", "app//com.couchbase.client.dcp.conductor.Fixer.run(Fixer.java:77)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 21, "id" : 82, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 350168564, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1320890684, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1044337796, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-4", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 95, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f1daf5a", "name" : "HttpExecutor(port:9641)-15", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 96, "name" : "threadDeathWatcher-3-1", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//com.couchbase.client.deps.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 8, "id" : 99, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@6d8bd307", "name" : "Executor-23:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 6, "id" : 100, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@6d8bd307", "name" : "Executor-24:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 8, "id" : 101, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@6d8bd307", "name" : "Executor-25:200758d4b55bbd40c0c91e8c2cc11473", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 24, "id" : 102, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16abc7", "name" : "FIFO Writer Thread", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.storage.common.buffercache.AsyncFIFOPageQueueManager.run(AsyncFIFOPageQueueManager.java:163)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 103, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 397625384, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1408021806, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1624000447, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-6", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 104, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 105, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 1, "id" : 106, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1429311730, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1489924212, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1857940358, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-7", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 107, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 108, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 3, "id" : 109, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1817098321, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 899748650, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 508008506, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-8", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 110, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-6", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 111, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-7", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 112, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-8", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 113, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-9", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 114, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-10", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 115, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-11", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 116, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-12", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 117, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-13", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 118, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-14", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 119, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@68a40c53", "name" : "HttpExecutor(port:9630)-15", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" } ] } }, { "config" : { "activeMemoryGlobalBudget" : 67108864, "activeStopTimeout" : 3600, "activeSuspendTimeout" : 3600, "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpAddress" : "127.0.0.1", "analyticsCcHttpPort" : 9601, "analyticsHttpAdminListenAddress" : "0.0.0.0", "analyticsHttpAdminListenPort" : 9626, "analyticsHttpAdminPublicAddress" : "127.0.0.1", "analyticsHttpAdminPublicPort" : 9626, "analyticsHttpAdminRequestQueueSize" : 32, "analyticsHttpAdminThreadCount" : 4, "analyticsHttpListenAddress" : "0.0.0.0", "analyticsHttpListenPort" : 9615, "analyticsHttpPublicAddress" : "127.0.0.1", "analyticsHttpPublicPort" : 9615, "analyticsHttpRequestQueueSize" : 256, "analyticsHttpThreadCount" : 16, "analyticsHttpsListenPort" : 19601, "analyticsNodeName" : "127.0.0.1:9001", "analyticsSslEnabled" : true, "bucketConfigAttemptTimeout" : 15, "bucketConfigTotalTimeout" : 60, "cbasAddress" : "127.0.0.1", "cbasPort" : 9625, "clusterAddress" : "127.0.0.1", "clusterConnectRetries" : 5, "clusterListenAddress" : "0.0.0.0", "clusterListenPort" : 9620, "clusterPort" : 9602, "clusterPublicAddress" : "127.0.0.1", "clusterPublicPort" : 9620, "collectDcpStateFromNodesTimeout" : 60, "compilerFramesize" : 32768, "compilerGroupmemory" : 33554432, "compilerJoinmemory" : 33554432, "compilerParallelism" : 0, "compilerSortmemory" : 33554432, "compilerStringoffset" : 0, "compilerTextsearchmemory" : 33554432, "coreDumpDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0/coredump", "dataListenAddress" : "0.0.0.0", "dataListenPort" : 9621, "dataPublicAddress" : "127.0.0.1", "dataPublicPort" : 9621, "dcpChannelConnectAttemptTimeout" : 15, "dcpChannelConnectTotalTimeout" : 60, "deadlockWatchdogHaltDelaySeconds" : 120, "deadlockWatchdogPollSeconds" : 300, "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "driverArgs" : [ "--jsonconfig", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0/config.json", "--nc", "-node-id", "d7a317d45ea35c6779cbb2ba8d1f7eac" ], "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_2" ], "jvmArgs" : null, "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_1", "logLevel" : "DEBUG", "maxWaitActiveCluster" : 60, "maxWebRequestSize" : 52428800, "messagingFrameCount" : 512, "messagingFrameSize" : 4096, "messagingListenAddress" : "0.0.0.0", "messagingListenPort" : 9623, "messagingPublicAddress" : "127.0.0.1", "messagingPublicPort" : 9623, "metadataCallbackPort" : 9629, "metadataListenPort" : 9628, "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "metadataRegistrationTimeoutSecs" : 60, "ncserviceAddress" : "127.0.0.1", "ncservicePid" : -1, "ncservicePort" : -1, "netBufferCount" : 1, "netThreadCount" : 1, "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9001, "publicAddress" : "127.0.0.1", "replicationListenAddress" : "0.0.0.0", "replicationListenPort" : 9627, "replicationPublicAddress" : "127.0.0.1", "replicationPublicPort" : 9627, "resultListenAddress" : "0.0.0.0", "resultListenPort" : 9622, "resultManagerMemory" : -1, "resultPublicAddress" : "127.0.0.1", "resultPublicPort" : 9622, "resultSweepThreshold" : 60000, "resultTtl" : 86400000, "startingPartitionId" : 3, "storageBuffercacheMaxopenfiles" : 2147483647, "storageBuffercachePagesize" : 131072, "storageBuffercacheSize" : 805306368, "storageLsmBloomfilterFalsepositiverate" : 0.01, "storageMaxActiveWritableDatasets" : 8, "storageMemorycomponentGlobalbudget" : 805306368, "storageMemorycomponentNumcomponents" : 2, "storageMemorycomponentPagesize" : 131072, "storageMetadataMemorycomponentNumpages" : 8, "threaddumpFrequencySeconds" : 300, "threaddumpLogLevel" : "DEBUG", "traceCategories" : [ ], "txnCommitprofilerEnabled" : false, "txnCommitprofilerReportinterval" : 5, "txnJobRecoveryMemorysize" : 67108864, "txnLockEscalationthreshold" : 1000, "txnLockShrinktimer" : 5000, "txnLockTimeoutSweepthreshold" : 10000, "txnLockTimeoutWaitthreshold" : 60000, "txnLogBufferNumpages" : 8, "txnLogBufferPagesize" : 262144, "txnLogCheckpointHistory" : 0, "txnLogCheckpointLsnthreshold" : 67108864, "txnLogCheckpointPollfrequency" : 120, "txnLogDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0/txn-log", "txnLogPartitionsize" : 268435456 }, "date" : "Sun Aug 12 14:33:29 PDT 2018", "metakv" : { "bootstrap/ensureCc/e8a78c35fcb3323712f05201c598bcde" : "MjAxOC0wOC0xMlQyMTozMjozMi4yOTE1MDda", "cluster/state/metadata/replica" : { "nodes" : [ { "location" : "127.0.0.1:9627", "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac" }, { "location" : "127.0.0.1:9642", "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473" } ], "revision" : 2 }, "config/node/200758d4b55bbd40c0c91e8c2cc11473" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9641", "analyticsHttpListenPort" : "9630", "analyticsHttpsListenPort" : "19602", "analyticsNodeName" : "127.0.0.1:9002", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9640", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9631", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9633", "ccClusterListenPort" : "9632", "ccConsoleListenPort" : "9634", "ccControllerId" : "2", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9635", "clusterPort" : "9602", "dataListenPort" : "9636", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_2", "messagingListenPort" : "9638", "metadataCallbackPort" : "9644", "metadataListenPort" : "9643", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9002, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9642, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9637", "startingPartitionId" : "6", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473" }, "config/node/d7a317d45ea35c6779cbb2ba8d1f7eac" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9626", "analyticsHttpListenPort" : "9615", "analyticsHttpsListenPort" : "19601", "analyticsNodeName" : "127.0.0.1:9001", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9625", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9616", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9618", "ccClusterListenPort" : "9617", "ccConsoleListenPort" : "9619", "ccControllerId" : "1", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9620", "clusterPort" : "9602", "dataListenPort" : "9621", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_1", "messagingListenPort" : "9623", "metadataCallbackPort" : "9629", "metadataListenPort" : "9628", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9001, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9627, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9622", "startingPartitionId" : "3", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac" }, "config/node/e8a78c35fcb3323712f05201c598bcde" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9611", "analyticsHttpListenPort" : "9600", "analyticsHttpsListenPort" : "19600", "analyticsNodeName" : "127.0.0.1:9000", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9610", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9601", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9603", "ccClusterListenPort" : "9602", "ccConsoleListenPort" : "9604", "ccControllerId" : "0", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9605", "clusterPort" : "9602", "dataListenPort" : "9606", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_0", "messagingListenPort" : "9608", "metadataCallbackPort" : "9614", "metadataListenPort" : "9613", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9000, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9612, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9607", "startingPartitionId" : "0", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "e8a78c35fcb3323712f05201c598bcde" }, "nextControllerId" : 3, "nextPartitionId" : 9, "topology" : { "balanceState" : "balanced", "ccNodeId" : "e8a78c35fcb3323712f05201c598bcde", "configVersion" : 1, "id" : "147bdd4381a8eb541c73daabddf1d333", "keepNodesUpdated" : false, "metadataNodeId" : "e8a78c35fcb3323712f05201c598bcde", "metadataPartition" : 0, "nodes" : [ { "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "opaque" : { "cc-http-port" : "9601", "controller-id" : "0", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "0" }, "priority" : 0 }, { "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "opaque" : { "cc-http-port" : "9616", "controller-id" : "1", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "3" }, "priority" : 0 }, { "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "opaque" : { "cc-http-port" : "9631", "controller-id" : "2", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "6" }, "priority" : 0 } ], "rev" : 2, "type" : "topology-change-rebalance" } }, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "nodeName" : "127.0.0.1:9001", "runtime" : { "bootDate" : "Sun Aug 12 14:31:54 PDT 2018", "daysWithoutAnAccident" : "0.00", "inputArguments" : [ "-XX:+ExitOnOutOfMemoryError", "-XX:MaxGCPauseMillis=60000", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9624", "-Xmx3072m", "-Dfile.encoding=UTF-8", "-Djava.net.preferIPv4Stack=true", "-Djava.net.preferIPv6Addresses=false", "--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED", "--add-opens=java.management/sun.management=ALL-UNNAMED", "-DlogDir=/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_1" ], "numProcessors" : 4, "pid" : 21455, "systemProperties" : { "awt.toolkit" : "sun.awt.X11.XToolkit", "com.couchbase.allowReverseDns" : "false", "com.couchbase.forceIPv4" : "false", "file.encoding" : "UTF-8", "file.encoding.pkg" : "sun.io", "file.separator" : "/", "java.awt.graphicsenv" : "sun.awt.X11GraphicsEnvironment", "java.awt.printerjob" : "sun.print.PSPrinterJob", "java.class.path" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-compiler.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-core.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-data.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-rewriter.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-runtime.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/args4j-2.33.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-active.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-algebra.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-app.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-external-data.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-aql.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-sqlpp.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-metadata.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-om.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-replication.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-runtime.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-transactions.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-connector.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-dcp-client.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-server.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-codec-1.9.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-collections4-4.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-compress-1.12.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-io-2.6.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-lang3-3.7.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-logging-1.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-text-1.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/core-io-1.5.6.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/disruptor-1.2.13-jdk8.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/esri-geometry-api-2.0.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/fastutil-8.1.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/guava-18.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/httpclient-4.5.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/httpcore-4.4.5.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-api.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-client.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-comm.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-cc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-nc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-data-std.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-dataflow-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-dataflow-std.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-http.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-ipc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-net.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-bloomfilter.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-btree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-btree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-invertedindex.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-rtree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-rtree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-util.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/ini4j-0.5.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-annotations-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-core-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-databind-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-api-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-core-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-jcl-2.11.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-jul-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-slf4j-impl-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-web-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/netty-all-4.1.25.Final.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/netty-tcnative-boringssl-static-2.0.8.Final.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/rxjava-1.3.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/slf4j-api-1.8.0-alpha2.jar" ], "java.class.version" : "54.0", "java.home" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/.jre/jre-10.0.2", "java.io.tmpdir" : "/tmp", "java.library.path" : [ "/usr/java/packages/lib", "/usr/lib64", "/lib64", "/lib", "/usr/lib" ], "java.net.preferIPv4Stack" : "true", "java.net.preferIPv6Addresses" : "false", "java.rmi.server.hostname" : "127.0.0.1", "java.runtime.name" : "Java(TM) SE Runtime Environment", "java.runtime.version" : "10.0.2+13", "java.specification.name" : "Java Platform API Specification", "java.specification.vendor" : "Oracle Corporation", "java.specification.version" : "10", "java.util.logging.manager" : "org.apache.logging.log4j.jul.LogManager", "java.vendor" : "Oracle Corporation", "java.vendor.url" : "http://java.oracle.com/", "java.vendor.url.bug" : "http://bugreport.java.com/bugreport/", "java.vendor.version" : "18.3", "java.version" : "10.0.2", "java.version.date" : "2018-07-17", "java.vm.compressedOopsMode" : "Zero based", "java.vm.info" : "mixed mode", "java.vm.name" : "Java HotSpot(TM) 64-Bit Server VM", "java.vm.specification.name" : "Java Virtual Machine Specification", "java.vm.specification.vendor" : "Oracle Corporation", "java.vm.specification.version" : "10", "java.vm.vendor" : "\"Oracle Corporation\"", "java.vm.version" : "10.0.2+13", "jdk.debug" : "release", "line.separator" : "\n", "log4j2.shutdownHookEnabled" : "false", "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_1", "os.arch" : "amd64", "os.name" : "Linux", "os.version" : "4.4.0-104-generic", "path.separator" : ":", "sun.arch.data.model" : "64", "sun.boot.library.path" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/.jre/jre-10.0.2/lib" ], "sun.cpu.endian" : "little", "sun.cpu.isalist" : "", "sun.io.unicode.encoding" : "UnicodeLittle", "sun.java.command" : [ "com.couchbase.analytics.control.AnalyticsDriver", "--jsonconfig", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0/config.json", "--nc", "-node-id", "d7a317d45ea35c6779cbb2ba8d1f7eac" ], "sun.java.launcher" : "SUN_STANDARD", "sun.jnu.encoding" : "UTF-8", "sun.management.compiler" : "HotSpot 64-Bit Tiered Compilers", "sun.nio.ch.bugLevel" : "", "sun.os.patch.level" : "unknown", "user.country" : "US", "user.dir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server", "user.home" : "/home/couchbase", "user.language" : "en", "user.name" : "couchbase", "user.timezone" : "America/Los_Angeles" }, "uptimeMillis" : 95444 }, "stats" : { "applicationCpuCoreBudget" : 4, "applicationMemoryBudget" : 1610612736, "date" : 1534109609528, "diskRead" : 0, "diskWrite" : 11390976, "gcs" : [ { "collection_count" : null, "collection_time" : null, "name" : "G1 Young Generation" }, { "collection_count" : null, "collection_time" : null, "name" : "G1 Old Generation" } ], "heapCommittedSize" : 264241152, "heapInitSize" : 264241152, "heapMaxSize" : 3221225472, "heapUsedSize" : 75757920, "heartbeatTime" : 1534109581837, "ipcMessageBytesReceived" : 173102, "ipcMessageBytesSent" : 237823, "ipcMessagesReceived" : 33, "ipcMessagesSent" : 38, "netPayloadBytesRead" : 16613916, "netPayloadBytesWritten" : 6849052, "netSignalingBytesRead" : 7072, "netSignalingBytesWritten" : 7192, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "nonheapCommittedSize" : 79101952, "nonheapInitSize" : 7667712, "nonheapMaxSize" : -1, "nonheapUsedSize" : 75870120, "peakThreadCount" : 105, "resultNetPayloadBytesRead" : 0, "resultNetPayloadBytesWritten" : 0, "resultNetSignalingBytesRead" : 0, "resultNetSignalingBytesWritten" : 0, "systemLoadAverage" : 6.49, "threadCount" : 105 }, "threaddump" : { "date" : "Sun Aug 12 14:33:29 PDT 2018", "threads" : [ { "blocked_count" : 10, "id" : 1, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@23a03e63", "name" : "main", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquire(Unknown Source)", "app//com.couchbase.analytics.control.AnalyticsDriver.main(AnalyticsDriver.java:111)" ], "state" : "WAITING" }, { "id" : 2, "name" : "Reference Handler", "stack" : [ "java.base@10.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method)", "java.base@10.0.2/java.lang.ref.Reference.processPendingReferences(Unknown Source)", "java.base@10.0.2/java.lang.ref.Reference.access$000(Unknown Source)", "java.base@10.0.2/java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 4, "id" : 3, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@2c8d444f", "name" : "Finalizer", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 4, "name" : "Signal Dispatcher", "stack" : [ ], "state" : "RUNNABLE" }, { "blocked_count" : 4, "id" : 9, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@2f86fa4f", "name" : "Common-Cleaner", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/jdk.internal.ref.CleanerImpl.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)", "java.base@10.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 10, "name" : "JDWP Transport Listener: dt_socket", "stack" : [ ], "state" : "RUNNABLE" }, { "id" : 11, "name" : "JDWP Event Helper Thread", "stack" : [ ], "state" : "RUNNABLE" }, { "id" : 14, "name" : "AsyncAppender-AsyncDebugLog", "stack" : [ "java.base@10.0.2/java.lang.Thread.yield(Native Method)", "app//com.conversantmedia.util.concurrent.Condition.progressiveYield(Condition.java:68)", "app//com.conversantmedia.util.concurrent.AbstractWaitingCondition.await(AbstractWaitingCondition.java:144)", "app//com.conversantmedia.util.concurrent.DisruptorBlockingQueue.take(DisruptorBlockingQueue.java:245)", "app//org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:403)" ], "state" : "RUNNABLE" }, { "id" : 16, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@48dc17d0", "name" : "ShutdownWatchdog", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquireUninterruptibly(Unknown Source)", "app//org.apache.hyracks.util.ExitUtil$ShutdownWatchdog.run(ExitUtil.java:109)" ], "state" : "WAITING" }, { "id" : 18, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a2d6b8e", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 501951555 } ], "name" : "IoRequestHandler-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 19, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a2d6b8e", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1002041276 } ], "name" : "IoRequestHandler-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 20, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a2d6b8e", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 485475507 } ], "name" : "IoRequestHandler-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 21, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a2d6b8e", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 724028528 } ], "name" : "IoRequestHandler-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 22, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a2d6b8e", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1869247587 } ], "name" : "IoRequestHandler-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 23, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a2d6b8e", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 467313213 } ], "name" : "IoRequestHandler-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 2, "id" : 25, "lock_name" : "java.util.TaskQueue@2cc4e5b8", "name" : "Timer-0", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.util.TimerThread.mainLoop(Unknown Source)", "java.base@10.0.2/java.util.TimerThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 26, "locked_monitors" : [ { "className" : "java.io.BufferedInputStream", "identityHashCode" : 872949485, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "java.io.BufferedInputStream", "fileName" : null, "lineNumber" : -1, "methodName" : "read", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "Stdin Watcher", "stack" : [ "java.base@10.0.2/java.io.FileInputStream.readBytes(Native Method)", "java.base@10.0.2/java.io.FileInputStream.read(Unknown Source)", "java.base@10.0.2/java.io.BufferedInputStream.fill(Unknown Source)", "java.base@10.0.2/java.io.BufferedInputStream.read(Unknown Source)", "app//com.couchbase.analytics.control.StdInWatcher.lambda$start$0(StdInWatcher.java:38)", "app//com.couchbase.analytics.control.StdInWatcher$$Lambda$140/1802736936.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 20, "id" : 27, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 482530603, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 489860134, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 374367005, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:9620]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "blocked_count" : 77, "id" : 28, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1662648436, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1778121393, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1629501492, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9621]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "id" : 29, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@747de786", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2081952435 } ], "name" : "Executor-1:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.transaction.management.service.logging.LogManager$FlushLogsLogger.run(LogManager.java:644)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 16, "id" : 30, "lock_name" : "org.apache.asterix.transaction.management.service.logging.LogBuffer@475740dd", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 13001549 } ], "name" : "Executor-2:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.Object.wait(Unknown Source)", "app//org.apache.asterix.transaction.management.service.logging.LogBuffer.flush(LogBuffer.java:176)", "app//org.apache.asterix.transaction.management.service.logging.LogFlusher.call(LogManager.java:699)", "app//org.apache.asterix.transaction.management.service.logging.LogFlusher.call(LogManager.java:654)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 31, "lock_name" : "org.apache.asterix.replication.logging.ReplicationLogBuffer@71cf04f4", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1591683105 } ], "name" : "TxnLog Replicator", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.Object.wait(Unknown Source)", "app//org.apache.asterix.replication.logging.ReplicationLogBuffer.flush(ReplicationLogBuffer.java:94)", "app//org.apache.asterix.replication.logging.TxnLogReplicator.call(TxnLogReplicator.java:93)", "app//org.apache.asterix.replication.logging.TxnLogReplicator.call(TxnLogReplicator.java:31)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 32, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11ce7361", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1090057640 } ], "name" : "ReplicationJobsProcessor", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.replication.management.IndexReplicationManager$ReplicationJobsProcessor.run(IndexReplicationManager.java:174)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 34, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@a94f709", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1779219567 } ], "name" : "d7a317d45ea35c6779cbb2ba8d1f7eacRemoteLogsNotifier", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.replication.logging.RemoteLogsNotifier.run(RemoteLogsNotifier.java:63)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 35, "lock_name" : "java.lang.Object@77b81f2e", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1501716646 } ], "name" : "Executor-6:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "app//org.apache.hyracks.storage.common.buffercache.DelayPageCleanerPolicy.notifyCleanCycleFinish(DelayPageCleanerPolicy.java:34)", "app//org.apache.hyracks.storage.common.buffercache.BufferCache$CleanerThread.runCleanCycle(BufferCache.java:773)", "app//org.apache.hyracks.storage.common.buffercache.BufferCache$CleanerThread.run(BufferCache.java:745)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 36, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e594d89", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 890733699 } ], "name" : "Executor-7:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.messaging.NCMessageBroker$MessageDeliveryService.run(NCMessageBroker.java:162)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 37, "lock_name" : "com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor@1cee7c58", "name" : "Thread-3", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "app//com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor.run(NsServerHttpClient.java:66)" ], "state" : "TIMED_WAITING" }, { "id" : 38, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1203638171 } ], "name" : "Executor-8:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//org.apache.hyracks.control.common.result.ResultStateSweeper.run(ResultStateSweeper.java:44)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 39, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 837807219, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 110740495, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 848884816, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9622]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "id" : 40, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1576747751, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 322548063, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1531491979, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9623]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "id" : 41, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@2e87e3c6", "name" : "RMI RenewClean-[127.0.0.1:9614]", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 42, "lock_name" : "sun.rmi.transport.GC$LatencyLock@29ec708b", "name" : "RMI GC Daemon", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.rmi@10.0.2/sun.rmi.transport.GC$Daemon.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)", "java.base@10.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 43, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77819fa0", "name" : "RMI Scheduler(0)", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 1, "id" : 44, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@69e38335", "name" : "d7a317d45ea35c6779cbb2ba8d1f7eac-Heartbeat", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.tryAcquire(Unknown Source)", "app//org.apache.hyracks.control.nc.heartbeat.HeartbeatTask.run(HeartbeatTask.java:59)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 14, "id" : 24, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@786bd41c", "name" : "Worker:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:114)" ], "state" : "WAITING" }, { "blocked_count" : 3, "id" : 47, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 852069298, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1238265556, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1019328243, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:38916]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "blocked_count" : 1, "id" : 48, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1aac5e43", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 585878713 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileRequestHandler", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileRequestHandler.run(HyracksConnection.java:431)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 49, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1517677623 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileHandlerWatcher", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "java.base@10.0.2/java.lang.Thread.sleep(Unknown Source)", "java.base@10.0.2/java.util.concurrent.TimeUnit.sleep(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileHandlerWatcher.run(HyracksConnection.java:459)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 51, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@6446d2bf", "name" : "ObjectCleanerThread", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "app//io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 52, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1589946413, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1862531880, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 388366029, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 33, "locked_monitors" : [ { "className" : "java.lang.Object", "identityHashCode" : 1168381372, "lockedStackDepth" : 2, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.ServerSocketChannelImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "accept", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "d7a317d45ea35c6779cbb2ba8d1f7eac Replication Channel Thread", "stack" : [ "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)", "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)", "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)", "app//org.apache.asterix.replication.management.ReplicationChannel.run(ReplicationChannel.java:72)" ], "state" : "RUNNABLE" }, { "id" : 53, "name" : "Checkpoint Thread (53)", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//org.apache.asterix.transaction.management.service.recovery.CheckpointThread.run(CheckpointThread.java:61)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 1, "id" : 54, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1046104293, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1315978941, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1981510184, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 7, "id" : 55, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 334991522, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2139549529, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 818721730, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-2", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 56, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 59, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 60, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 61, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 850852942, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2046959974, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 797302599, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-3", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 62, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1791318627, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1863618553, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 74877548, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-4", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 63, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 6, "id" : 64, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 92799554, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1172199122, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 251556567, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-5", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 5, "id" : 65, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1055636622, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2067018499, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1290506865, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-6", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 1, "id" : 66, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 830351857, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1376971932, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 140796406, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-7", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 67, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 68, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 69, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 70, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 187, "id" : 71, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1743562686 } ], "name" : "Executor-13:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.management@10.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)", "java.management@10.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)", "java.management@10.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)", "app//org.apache.hyracks.util.ThreadDumpUtil.takeDumpJSON(ThreadDumpUtil.java:57)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$getResultJson$0(NodeDiagnosticsServlet.java:86)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$357/2027401776.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 1, "id" : 72, "locked_monitors" : [ { "className" : "org.apache.hyracks.control.common.NodeControllerData", "identityHashCode" : 1484620240, "lockedStackDepth" : 16, "lockedStackFrame" : { "classLoaderName" : "app", "className" : "org.apache.hyracks.control.common.NodeControllerData", "fileName" : "NodeControllerData.java", "lineNumber" : 314, "methodName" : "toDetailedJSON", "moduleName" : null, "moduleVersion" : null, "nativeMethod" : false } } ], "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 710024481 } ], "name" : "Executor-14:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.defineAnonymousClass0(Native Method)", "java.base@10.0.2/jdk.internal.misc.Unsafe.defineAnonymousClass(Unknown Source)", "java.base@10.0.2/java.lang.invoke.InvokerBytecodeGenerator.loadAndInitializeInvokerClass(Unknown Source)", "java.base@10.0.2/java.lang.invoke.InvokerBytecodeGenerator.loadMethod(Unknown Source)", "java.base@10.0.2/java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(Unknown Source)", "java.base@10.0.2/java.lang.invoke.LambdaForm.compileToBytecode(Unknown Source)", "java.base@10.0.2/java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(Unknown Source)", "java.base@10.0.2/java.lang.invoke.DirectMethodHandle.preparedLambdaForm(Unknown Source)", "java.base@10.0.2/java.lang.invoke.DirectMethodHandle.preparedLambdaForm(Unknown Source)", "java.base@10.0.2/java.lang.invoke.DirectMethodHandle.make(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(Unknown Source)", "app//org.apache.hyracks.util.JSONUtil.put(JSONUtil.java:253)", "app//org.apache.hyracks.control.common.NodeControllerData.toDetailedJSON(NodeControllerData.java:314)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.processNodeStats(NodeDiagnosticsServlet.java:129)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$getResultJson$2(NodeDiagnosticsServlet.java:89)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$359/971683851.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 3, "id" : 73, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1250795173 } ], "name" : "Executor-15:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.defineAnonymousClass0(Native Method)", "java.base@10.0.2/jdk.internal.misc.Unsafe.defineAnonymousClass(Unknown Source)", "java.base@10.0.2/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(Unknown Source)", "java.base@10.0.2/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(Unknown Source)", "java.base@10.0.2/java.lang.invoke.LambdaMetafactory.metafactory(Unknown Source)", "java.base@10.0.2/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(Unknown Source)", "java.base@10.0.2/java.lang.invoke.LambdaForm$MH/1009326765.invoke(LambdaForm$MH)", "java.base@10.0.2/java.lang.invoke.Invokers$Holder.invoke_MT(Unknown Source)", "java.base@10.0.2/java.lang.invoke.CallSite.makeSite(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(Unknown Source)", "java.base@10.0.2/java.lang.invoke.MethodHandleNatives.linkCallSite(Unknown Source)", "app//org.apache.hyracks.control.common.config.OptionTypes$10.serializeJSONField(OptionTypes.java:312)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$populateConfig$4(NodeDiagnosticsServlet.java:164)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$361/2053821671.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)", "java.base@10.0.2/java.util.HashMap$KeySpliterator.forEachRemaining(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.copyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.evaluate(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline.forEach(Unknown Source)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.populateConfig(NodeDiagnosticsServlet.java:164)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$getResultJson$1(NodeDiagnosticsServlet.java:87)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$358/2095593970.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 278, "id" : 74, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@1dcee04f", "name" : "Executor-16:d7a317d45ea35c6779cbb2ba8d1f7eac", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 76, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 342078563 } ], "name" : "HttpExecutor(port:9626)-6", "stack" : [ "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.createSupportedLocaleString(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.createLanguageTagSet(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.getLanguageTagSet(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.lambda$getTimeZoneNameProvider$8(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter$$Lambda$362/897817375.run(Unknown Source)", "java.base@10.0.2/java.security.AccessController.doPrivileged(Native Method)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.getTimeZoneNameProvider(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.JRELocaleProviderAdapter.getLocaleServiceProvider(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.LocaleServiceProviderPool.findProviders(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObjectImpl(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.LocaleServiceProviderPool.getLocalizedObject(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayNamesImpl(Unknown Source)", "java.base@10.0.2/sun.util.locale.provider.TimeZoneNameUtility.retrieveDisplayName(Unknown Source)", "java.base@10.0.2/java.util.TimeZone.getDisplayName(Unknown Source)", "java.base@10.0.2/java.util.Date.toString(Unknown Source)", "java.base@10.0.2/java.lang.String.valueOf(Unknown Source)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.getResultJson(NodeDiagnosticsServlet.java:100)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.get(NodeDiagnosticsServlet.java:74)", "app//org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:88)", "app//com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:79)", "app//org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:71)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:56)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 77, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-7", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 78, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-8", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 79, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-9", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 80, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-10", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 81, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-11", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 1, "id" : 91, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-12", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 69, "id" : 83, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 906910161, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1315884315, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 2049549831, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-2", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 61, "id" : 84, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 277867925, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1418456708, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1815475995, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-3", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 94, "id" : 82, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 1536912644, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 882360487, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1063511935, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 93, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-13", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 94, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-14", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 95, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d6b1dc1", "name" : "HttpExecutor(port:9626)-15", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 17, "id" : 85, "locked_monitors" : [ { "className" : "com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySet", "identityHashCode" : 2091733671, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 2094986107, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1984337599, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-4", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//com.couchbase.client.deps.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:754)", "app//com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)", "app//com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 99, "name" : "threadDeathWatcher-3-1", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//com.couchbase.client.deps.io.netty.util.ThreadDeathWatcher$Watcher.run(ThreadDeathWatcher.java:152)", "app//com.couchbase.client.deps.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 24, "id" : 106, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56bedf17", "name" : "FIFO Writer Thread", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.storage.common.buffercache.AsyncFIFOPageQueueManager.run(AsyncFIFOPageQueueManager.java:163)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 107, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1074829069, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1272901983, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 158299637, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-3-8", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 108, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 4, "id" : 109, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 110, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 111, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 112, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-6", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 113, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-7", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 114, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-8", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 115, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-9", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 116, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-10", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 117, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-11", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 118, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-12", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 119, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-13", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 120, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-14", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 121, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188c1291", "name" : "HttpExecutor(port:9615)-15", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" } ] } }, { "cc" : { "config" : { "address" : "0.0.0.0", "analyticsHttpListenAddress" : "0.0.0.0", "analyticsHttpListenPort" : 9601, "analyticsMetadataPartitionId" : 0, "clientListenAddress" : "0.0.0.0", "clientListenPort" : 9603, "clientPublicAddress" : "127.0.0.1", "clientPublicPort" : 9603, "clusterListenAddress" : "0.0.0.0", "clusterListenPort" : 9602, "clusterPublicAddress" : "127.0.0.1", "clusterPublicPort" : 9602, "consoleListenAddress" : "0.0.0.0", "consoleListenPort" : 9604, "consolePublicAddress" : "127.0.0.1", "consolePublicPort" : 9604, "controllerId" : 0, "coresMultiplier" : 3, "enforceFrameWriterProtocol" : false, "heartbeatMaxMisses" : 4, "heartbeatPeriod" : 30000, "jobHistorySize" : 10, "jobQueueCapacity" : 4096, "profileDumpPeriod" : 0, "publicAddress" : "127.0.0.1", "resultSweepThreshold" : 60000, "resultTtl" : 86400000, "rootDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0/ClusterControllerService" }, "partitions" : [ { "active" : true, "activeNodeId" : "e8a78c35fcb3323712f05201c598bcde", "iodeviceNum" : 0, "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "partitionId" : 0, "pendingActivation" : false }, { "active" : true, "activeNodeId" : "e8a78c35fcb3323712f05201c598bcde", "iodeviceNum" : 1, "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "partitionId" : 1, "pendingActivation" : false }, { "active" : true, "activeNodeId" : "e8a78c35fcb3323712f05201c598bcde", "iodeviceNum" : 2, "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "partitionId" : 2, "pendingActivation" : false }, { "active" : true, "activeNodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "iodeviceNum" : 0, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "partitionId" : 3, "pendingActivation" : false }, { "active" : true, "activeNodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "iodeviceNum" : 1, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "partitionId" : 4, "pendingActivation" : false }, { "active" : true, "activeNodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "iodeviceNum" : 2, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "partitionId" : 5, "pendingActivation" : false }, { "active" : false, "activeNodeId" : null, "iodeviceNum" : 0, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "partitionId" : 6, "pendingActivation" : false }, { "active" : false, "activeNodeId" : null, "iodeviceNum" : 1, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "partitionId" : 7, "pendingActivation" : false }, { "active" : false, "activeNodeId" : null, "iodeviceNum" : 2, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "partitionId" : 8, "pendingActivation" : false } ], "state" : "UNUSABLE" }, "config" : { "activeMemoryGlobalBudget" : 67108864, "activeStopTimeout" : 3600, "activeSuspendTimeout" : 3600, "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpAddress" : "127.0.0.1", "analyticsCcHttpPort" : 9601, "analyticsHttpAdminListenAddress" : "0.0.0.0", "analyticsHttpAdminListenPort" : 9611, "analyticsHttpAdminPublicAddress" : "127.0.0.1", "analyticsHttpAdminPublicPort" : 9611, "analyticsHttpAdminRequestQueueSize" : 32, "analyticsHttpAdminThreadCount" : 4, "analyticsHttpListenAddress" : "0.0.0.0", "analyticsHttpListenPort" : 9600, "analyticsHttpPublicAddress" : "127.0.0.1", "analyticsHttpPublicPort" : 9600, "analyticsHttpRequestQueueSize" : 256, "analyticsHttpThreadCount" : 16, "analyticsHttpsListenPort" : 19600, "analyticsNodeName" : "127.0.0.1:9000", "analyticsSslEnabled" : true, "bucketConfigAttemptTimeout" : 15, "bucketConfigTotalTimeout" : 60, "cbasAddress" : "127.0.0.1", "cbasPort" : 9610, "clusterAddress" : "127.0.0.1", "clusterConnectRetries" : 5, "clusterListenAddress" : "0.0.0.0", "clusterListenPort" : 9605, "clusterPort" : 9602, "clusterPublicAddress" : "127.0.0.1", "clusterPublicPort" : 9605, "collectDcpStateFromNodesTimeout" : 60, "compilerFramesize" : 32768, "compilerGroupmemory" : 33554432, "compilerJoinmemory" : 33554432, "compilerParallelism" : 0, "compilerSortmemory" : 33554432, "compilerStringoffset" : 0, "compilerTextsearchmemory" : 33554432, "coreDumpDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0/coredump", "dataListenAddress" : "0.0.0.0", "dataListenPort" : 9606, "dataPublicAddress" : "127.0.0.1", "dataPublicPort" : 9606, "dcpChannelConnectAttemptTimeout" : 15, "dcpChannelConnectTotalTimeout" : 60, "deadlockWatchdogHaltDelaySeconds" : 120, "deadlockWatchdogPollSeconds" : 300, "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "driverArgs" : [ "--jsonconfig", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0/config.json", "--nc", "-node-id", "e8a78c35fcb3323712f05201c598bcde", "--cc" ], "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_2" ], "jvmArgs" : null, "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_0", "logLevel" : "DEBUG", "maxWaitActiveCluster" : 60, "maxWebRequestSize" : 52428800, "messagingFrameCount" : 512, "messagingFrameSize" : 4096, "messagingListenAddress" : "0.0.0.0", "messagingListenPort" : 9608, "messagingPublicAddress" : "127.0.0.1", "messagingPublicPort" : 9608, "metadataCallbackPort" : 9614, "metadataListenPort" : 9613, "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "metadataRegistrationTimeoutSecs" : 60, "ncserviceAddress" : "127.0.0.1", "ncservicePid" : -1, "ncservicePort" : -1, "netBufferCount" : 1, "netThreadCount" : 1, "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9000, "publicAddress" : "127.0.0.1", "replicationListenAddress" : "0.0.0.0", "replicationListenPort" : 9612, "replicationPublicAddress" : "127.0.0.1", "replicationPublicPort" : 9612, "resultListenAddress" : "0.0.0.0", "resultListenPort" : 9607, "resultManagerMemory" : -1, "resultPublicAddress" : "127.0.0.1", "resultPublicPort" : 9607, "resultSweepThreshold" : 60000, "resultTtl" : 86400000, "startingPartitionId" : 0, "storageBuffercacheMaxopenfiles" : 2147483647, "storageBuffercachePagesize" : 131072, "storageBuffercacheSize" : 805306368, "storageLsmBloomfilterFalsepositiverate" : 0.01, "storageMaxActiveWritableDatasets" : 8, "storageMemorycomponentGlobalbudget" : 805306368, "storageMemorycomponentNumcomponents" : 2, "storageMemorycomponentPagesize" : 131072, "storageMetadataMemorycomponentNumpages" : 8, "threaddumpFrequencySeconds" : 300, "threaddumpLogLevel" : "DEBUG", "traceCategories" : [ ], "txnCommitprofilerEnabled" : false, "txnCommitprofilerReportinterval" : 5, "txnJobRecoveryMemorysize" : 67108864, "txnLockEscalationthreshold" : 1000, "txnLockShrinktimer" : 5000, "txnLockTimeoutSweepthreshold" : 10000, "txnLockTimeoutWaitthreshold" : 60000, "txnLogBufferNumpages" : 8, "txnLogBufferPagesize" : 262144, "txnLogCheckpointHistory" : 0, "txnLogCheckpointLsnthreshold" : 67108864, "txnLogCheckpointPollfrequency" : 120, "txnLogDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0/txn-log", "txnLogPartitionsize" : 268435456 }, "date" : "Sun Aug 12 14:33:29 PDT 2018", "metakv" : { "bootstrap/ensureCc/e8a78c35fcb3323712f05201c598bcde" : "MjAxOC0wOC0xMlQyMTozMjozMi4yOTE1MDda", "cluster/state/metadata/replica" : { "nodes" : [ { "location" : "127.0.0.1:9627", "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac" }, { "location" : "127.0.0.1:9642", "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473" } ], "revision" : 2 }, "config/node/200758d4b55bbd40c0c91e8c2cc11473" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9641", "analyticsHttpListenPort" : "9630", "analyticsHttpsListenPort" : "19602", "analyticsNodeName" : "127.0.0.1:9002", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9640", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9631", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9633", "ccClusterListenPort" : "9632", "ccConsoleListenPort" : "9634", "ccControllerId" : "2", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9635", "clusterPort" : "9602", "dataListenPort" : "9636", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_2/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_2", "messagingListenPort" : "9638", "metadataCallbackPort" : "9644", "metadataListenPort" : "9643", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9002, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9642, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9637", "startingPartitionId" : "6", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473" }, "config/node/d7a317d45ea35c6779cbb2ba8d1f7eac" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9626", "analyticsHttpListenPort" : "9615", "analyticsHttpsListenPort" : "19601", "analyticsNodeName" : "127.0.0.1:9001", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9625", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9616", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9618", "ccClusterListenPort" : "9617", "ccConsoleListenPort" : "9619", "ccControllerId" : "1", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9620", "clusterPort" : "9602", "dataListenPort" : "9621", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_1/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_1", "messagingListenPort" : "9623", "metadataCallbackPort" : "9629", "metadataListenPort" : "9628", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9001, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9627, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9622", "startingPartitionId" : "3", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac" }, "config/node/e8a78c35fcb3323712f05201c598bcde" : { "config" : { "additionalCcNodes" : [ ], "address" : "0.0.0.0", "analyticsCbHome" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/install", "analyticsCcHttpPort" : "9601", "analyticsHttpAdminListenPort" : "9611", "analyticsHttpListenPort" : "9600", "analyticsHttpsListenPort" : "19600", "analyticsNodeName" : "127.0.0.1:9000", "analyticsSslEnabled" : true, "cbasAddress" : "127.0.0.1", "cbasPort" : "9610", "ccAddress" : "0.0.0.0", "ccAnalyticsHttpListenPort" : "9601", "ccAnalyticsMetadataPartitionId" : 0, "ccClientListenPort" : "9603", "ccClusterListenPort" : "9602", "ccConsoleListenPort" : "9604", "ccControllerId" : "0", "ccHeartbeatMaxMisses" : "4", "ccHeartbeatPeriod" : "30000", "ccPublicAddress" : "127.0.0.1", "clusterAddress" : "127.0.0.1", "clusterListenPort" : "9605", "clusterPort" : "9602", "dataListenPort" : "9606", "defaultDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "initialRun" : false, "iodevices" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_1", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_2" ], "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_0", "messagingListenPort" : "9608", "metadataCallbackPort" : "9614", "metadataListenPort" : "9613", "metadataNode" : "e8a78c35fcb3323712f05201c598bcde", "ncservicePort" : "-1", "nsServerAddress" : "127.0.0.1", "nsServerPort" : 9000, "publicAddress" : "127.0.0.1", "replicationEnabled" : true, "replicationFactor" : 2, "replicationListenPort" : 9612, "replicationLogBufferPagesize" : 262144, "replicationStrategy" : "metadata", "resultListenPort" : "9607", "startingPartitionId" : "0", "txnLogBufferPagesize" : 262144 }, "configVersion" : 1, "nodeId" : "e8a78c35fcb3323712f05201c598bcde" }, "nextControllerId" : 3, "nextPartitionId" : 9, "topology" : { "balanceState" : "balanced", "ccNodeId" : "e8a78c35fcb3323712f05201c598bcde", "configVersion" : 1, "id" : "147bdd4381a8eb541c73daabddf1d333", "keepNodesUpdated" : false, "metadataNodeId" : "e8a78c35fcb3323712f05201c598bcde", "metadataPartition" : 0, "nodes" : [ { "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "opaque" : { "cc-http-port" : "9601", "controller-id" : "0", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "0" }, "priority" : 0 }, { "nodeId" : "d7a317d45ea35c6779cbb2ba8d1f7eac", "opaque" : { "cc-http-port" : "9616", "controller-id" : "1", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "3" }, "priority" : 0 }, { "nodeId" : "200758d4b55bbd40c0c91e8c2cc11473", "opaque" : { "cc-http-port" : "9631", "controller-id" : "2", "host" : "127.0.0.1", "num-iodevices" : "3", "starting-partition-id" : "6" }, "priority" : 0 } ], "rev" : 2, "type" : "topology-change-rebalance" } }, "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "nodeName" : "127.0.0.1:9000", "runtime" : { "bootDate" : "Sun Aug 12 14:32:24 PDT 2018", "daysWithoutAnAccident" : "0.00", "inputArguments" : [ "-XX:+ExitOnOutOfMemoryError", "-XX:MaxGCPauseMillis=60000", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9609", "-Xmx3072m", "-Dfile.encoding=UTF-8", "-Djava.net.preferIPv4Stack=true", "-Djava.net.preferIPv6Addresses=false", "--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED", "--add-opens=java.management/sun.management=ALL-UNNAMED", "-DlogDir=/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_0" ], "numProcessors" : 4, "pid" : 21807, "systemProperties" : { "awt.toolkit" : "sun.awt.X11.XToolkit", "com.couchbase.allowReverseDns" : "false", "com.couchbase.forceIPv4" : "false", "file.encoding" : "UTF-8", "file.encoding.pkg" : "sun.io", "file.separator" : "/", "java.awt.graphicsenv" : "sun.awt.X11GraphicsEnvironment", "java.awt.printerjob" : "sun.print.PSPrinterJob", "java.class.path" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-compiler.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-core.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-data.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-rewriter.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/algebricks-runtime.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/args4j-2.33.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-active.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-algebra.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-app.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-external-data.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-aql.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-lang-sqlpp.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-metadata.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-om.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-replication.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-runtime.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/asterix-transactions.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-connector.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-dcp-client.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/cbas-server.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-codec-1.9.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-collections4-4.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-compress-1.12.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-io-2.6.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-lang3-3.7.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-logging-1.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/commons-text-1.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/core-io-1.5.6.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/disruptor-1.2.13-jdk8.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/esri-geometry-api-2.0.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/fastutil-8.1.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/guava-18.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/httpclient-4.5.2.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/httpcore-4.4.5.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-api.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-client.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-comm.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-cc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-control-nc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-data-std.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-dataflow-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-dataflow-std.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-http.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-ipc.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-net.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-bloomfilter.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-btree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-btree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-invertedindex.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-lsm-rtree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-am-rtree.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-storage-common.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/hyracks-util.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/ini4j-0.5.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-annotations-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-core-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/jackson-databind-2.8.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-api-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-core-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-jcl-2.11.1.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-jul-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-slf4j-impl-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/log4j-web-2.10.0.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/netty-all-4.1.25.Final.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/netty-tcnative-boringssl-static-2.0.8.Final.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/rxjava-1.3.4.jar", "/home/couchbase/jenkins/workspace/cbas-jre10-test/install/lib/cbas/repo/slf4j-api-1.8.0-alpha2.jar" ], "java.class.version" : "54.0", "java.home" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/.jre/jre-10.0.2", "java.io.tmpdir" : "/tmp", "java.library.path" : [ "/usr/java/packages/lib", "/usr/lib64", "/lib64", "/lib", "/usr/lib" ], "java.net.preferIPv4Stack" : "true", "java.net.preferIPv6Addresses" : "false", "java.rmi.server.hostname" : "127.0.0.1", "java.runtime.name" : "Java(TM) SE Runtime Environment", "java.runtime.version" : "10.0.2+13", "java.specification.name" : "Java Platform API Specification", "java.specification.vendor" : "Oracle Corporation", "java.specification.version" : "10", "java.util.logging.manager" : "org.apache.logging.log4j.jul.LogManager", "java.vendor" : "Oracle Corporation", "java.vendor.url" : "http://java.oracle.com/", "java.vendor.url.bug" : "http://bugreport.java.com/bugreport/", "java.vendor.version" : "18.3", "java.version" : "10.0.2", "java.version.date" : "2018-07-17", "java.vm.compressedOopsMode" : "Zero based", "java.vm.info" : "mixed mode", "java.vm.name" : "Java HotSpot(TM) 64-Bit Server VM", "java.vm.specification.name" : "Java Virtual Machine Specification", "java.vm.specification.vendor" : "Oracle Corporation", "java.vm.specification.version" : "10", "java.vm.vendor" : "\"Oracle Corporation\"", "java.vm.version" : "10.0.2+13", "jdk.debug" : "release", "line.separator" : "\n", "log4j2.shutdownHookEnabled" : "false", "logDir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/logs/n_0", "os.arch" : "amd64", "os.name" : "Linux", "os.version" : "4.4.0-104-generic", "path.separator" : ":", "sun.arch.data.model" : "64", "sun.boot.library.path" : [ "/home/couchbase/jenkins/workspace/cbas-jre10-test/.jre/jre-10.0.2/lib" ], "sun.cpu.endian" : "little", "sun.cpu.isalist" : "", "sun.io.unicode.encoding" : "UnicodeLittle", "sun.java.command" : [ "com.couchbase.analytics.control.AnalyticsDriver", "--jsonconfig", "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir/@analytics/v_iodevice_0/config.json", "--nc", "-node-id", "e8a78c35fcb3323712f05201c598bcde", "--cc" ], "sun.java.launcher" : "SUN_STANDARD", "sun.jnu.encoding" : "UTF-8", "sun.management.compiler" : "HotSpot 64-Bit Tiered Compilers", "sun.nio.ch.bugLevel" : "", "sun.os.patch.level" : "unknown", "user.country" : "US", "user.dir" : "/home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server", "user.home" : "/home/couchbase", "user.language" : "en", "user.name" : "couchbase", "user.timezone" : "America/Los_Angeles" }, "uptimeMillis" : 66013 }, "stats" : { "applicationCpuCoreBudget" : 4, "applicationMemoryBudget" : 1610612736, "date" : 1534109609572, "diskRead" : 0, "diskWrite" : 65536, "gcs" : [ { "collection_count" : null, "collection_time" : null, "name" : "G1 Young Generation" }, { "collection_count" : null, "collection_time" : null, "name" : "G1 Old Generation" } ], "heapCommittedSize" : 264241152, "heapInitSize" : 264241152, "heapMaxSize" : 3221225472, "heapUsedSize" : 77185432, "heartbeatTime" : 1534109553855, "ipcMessageBytesReceived" : 0, "ipcMessageBytesSent" : 0, "ipcMessagesReceived" : 0, "ipcMessagesSent" : 0, "netPayloadBytesRead" : 0, "netPayloadBytesWritten" : 0, "netSignalingBytesRead" : 0, "netSignalingBytesWritten" : 0, "nodeId" : "e8a78c35fcb3323712f05201c598bcde", "nonheapCommittedSize" : 59240448, "nonheapInitSize" : 7667712, "nonheapMaxSize" : -1, "nonheapUsedSize" : 55284472, "peakThreadCount" : 50, "resultNetPayloadBytesRead" : 0, "resultNetPayloadBytesWritten" : 0, "resultNetSignalingBytesRead" : 0, "resultNetSignalingBytesWritten" : 0, "systemLoadAverage" : 8.19, "threadCount" : 50 }, "threaddump" : { "date" : "Sun Aug 12 14:33:29 PDT 2018", "threads" : [ { "blocked_count" : 23, "id" : 1, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@4dd3dc20", "name" : "main", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquire(Unknown Source)", "app//com.couchbase.analytics.control.AnalyticsDriver.main(AnalyticsDriver.java:111)" ], "state" : "WAITING" }, { "blocked_count" : 1, "id" : 2, "name" : "Reference Handler", "stack" : [ "java.base@10.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method)", "java.base@10.0.2/java.lang.ref.Reference.processPendingReferences(Unknown Source)", "java.base@10.0.2/java.lang.ref.Reference.access$000(Unknown Source)", "java.base@10.0.2/java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 3, "id" : 3, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@5f958060", "name" : "Finalizer", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 4, "name" : "Signal Dispatcher", "stack" : [ ], "state" : "RUNNABLE" }, { "blocked_count" : 3, "id" : 9, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@17485cbe", "name" : "Common-Cleaner", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/jdk.internal.ref.CleanerImpl.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)", "java.base@10.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 10, "name" : "JDWP Transport Listener: dt_socket", "stack" : [ ], "state" : "RUNNABLE" }, { "id" : 11, "name" : "JDWP Event Helper Thread", "stack" : [ ], "state" : "RUNNABLE" }, { "id" : 14, "name" : "AsyncAppender-AsyncDebugLog", "stack" : [ "java.base@10.0.2/java.lang.Thread.yield(Native Method)", "app//com.conversantmedia.util.concurrent.Condition.progressiveYield(Condition.java:68)", "app//com.conversantmedia.util.concurrent.AbstractWaitingCondition.await(AbstractWaitingCondition.java:144)", "app//com.conversantmedia.util.concurrent.DisruptorBlockingQueue.take(DisruptorBlockingQueue.java:245)", "app//org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run(AsyncAppender.java:403)" ], "state" : "RUNNABLE" }, { "id" : 16, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@5eaf1f90", "name" : "ShutdownWatchdog", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.acquireUninterruptibly(Unknown Source)", "app//org.apache.hyracks.util.ExitUtil$ShutdownWatchdog.run(ExitUtil.java:109)" ], "state" : "WAITING" }, { "id" : 18, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@dceeab9", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1121891386 } ], "name" : "IoRequestHandler-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 19, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@dceeab9", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 501951555 } ], "name" : "IoRequestHandler-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 20, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@dceeab9", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1002041276 } ], "name" : "IoRequestHandler-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 21, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@dceeab9", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 485475507 } ], "name" : "IoRequestHandler-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 22, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@dceeab9", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 724028528 } ], "name" : "IoRequestHandler-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 23, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@dceeab9", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1869247587 } ], "name" : "IoRequestHandler-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.nc.io.IoRequestHandler.run(IoRequestHandler.java:45)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 2, "id" : 25, "lock_name" : "java.util.TaskQueue@7eabede0", "name" : "Timer-0", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.util.TimerThread.mainLoop(Unknown Source)", "java.base@10.0.2/java.util.TimerThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 1, "id" : 27, "lock_name" : "java.util.TaskQueue@2b53e327", "name" : "Timer-1", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.util.TimerThread.mainLoop(Unknown Source)", "java.base@10.0.2/java.util.TimerThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 28, "locked_monitors" : [ { "className" : "java.io.BufferedInputStream", "identityHashCode" : 228993602, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "java.io.BufferedInputStream", "fileName" : null, "lineNumber" : -1, "methodName" : "read", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "Stdin Watcher", "stack" : [ "java.base@10.0.2/java.io.FileInputStream.readBytes(Native Method)", "java.base@10.0.2/java.io.FileInputStream.read(Unknown Source)", "java.base@10.0.2/java.io.BufferedInputStream.fill(Unknown Source)", "java.base@10.0.2/java.io.BufferedInputStream.read(Unknown Source)", "app//com.couchbase.analytics.control.StdInWatcher.lambda$start$0(StdInWatcher.java:38)", "app//com.couchbase.analytics.control.StdInWatcher$$Lambda$143/1248276879.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 9, "id" : 29, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 2009563757, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1445111192, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1658601120, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:9602]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "id" : 30, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1586818179, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 353098198, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1447473290, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:9603]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "id" : 31, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@556caddc", "name" : "ObjectCleanerThread", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "app//io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:54)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 32, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 744858374, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1743460917, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 71967906, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-2-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 33, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 880311226, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 897876896, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 167841175, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:41954]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "id" : 34, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fbb4741", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 663277822 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileRequestHandler", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileRequestHandler.run(HyracksConnection.java:431)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 35, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 79706679 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileHandlerWatcher", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "java.base@10.0.2/java.lang.Thread.sleep(Unknown Source)", "java.base@10.0.2/java.util.concurrent.TimeUnit.sleep(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileHandlerWatcher.run(HyracksConnection.java:459)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 36, "lock_name" : "com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor@286942a7", "name" : "Thread-5", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "app//com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor.run(NsServerHttpClient.java:66)" ], "state" : "TIMED_WAITING" }, { "id" : 37, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39e15d89", "name" : "ActiveNotificationHandler", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.api.util.SingleThreadEventProcessor.run(SingleThreadEventProcessor.java:49)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 38, "name" : "RMI TCP Accept-9614", "stack" : [ "java.base@10.0.2/java.net.PlainSocketImpl.socketAccept(Native Method)", "java.base@10.0.2/java.net.AbstractPlainSocketImpl.accept(Unknown Source)", "java.base@10.0.2/java.net.ServerSocket.implAccept(Unknown Source)", "java.base@10.0.2/java.net.ServerSocket.accept(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 39, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@3b0e8297", "name" : "RMI Reaper", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.ObjectTable$Reaper.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 40, "lock_name" : "sun.rmi.transport.GC$LatencyLock@3e0a5bf8", "name" : "RMI GC Daemon", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.rmi@10.0.2/sun.rmi.transport.GC$Daemon.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)", "java.base@10.0.2/jdk.internal.misc.InnocuousThread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 3, "id" : 41, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 17460152 } ], "name" : "Executor-1:ClusterController", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//org.apache.hyracks.control.common.result.ResultStateSweeper.run(ResultStateSweeper.java:44)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 42, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1544837860, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1865042426, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 553608370, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-4-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 43, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4c463ece", "name" : "Thread-6", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.hyracks.bootstrap.ClusterWorkExecutor.run(ClusterWorkExecutor.java:45)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 4, "id" : 26, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@327b9ed8", "name" : "Worker:ClusterController", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:114)" ], "state" : "WAITING" }, { "id" : 44, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@f2364f4", "name" : "RMI TCP Connection(idle)", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 6, "id" : 45, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1419922262, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 703577922, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1200388601, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:9605]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "id" : 46, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1402777912, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 354597094, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1105395887, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9606]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "blocked_count" : 2, "id" : 47, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@f2364f4", "name" : "RMI TCP Connection(idle)", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 48, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6584eadb", "name" : "RMI Scheduler(0)", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 49, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@53279222", "name" : "Executor-2:ClusterController", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 50, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@53279222", "name" : "Executor-3:ClusterController", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 51, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53c2f528", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 725884502 } ], "name" : "Executor-1:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.transaction.management.service.logging.LogManager$FlushLogsLogger.run(LogManager.java:644)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 36, "id" : 52, "lock_name" : "org.apache.asterix.transaction.management.service.logging.LogBuffer@afd6187", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 648346093 } ], "name" : "Executor-2:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.Object.wait(Unknown Source)", "app//org.apache.asterix.transaction.management.service.logging.LogBuffer.flush(LogBuffer.java:176)", "app//org.apache.asterix.transaction.management.service.logging.LogFlusher.call(LogManager.java:699)", "app//org.apache.asterix.transaction.management.service.logging.LogFlusher.call(LogManager.java:654)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 7, "id" : 53, "lock_name" : "org.apache.asterix.replication.logging.ReplicationLogBuffer@48ed9031", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1418286721 } ], "name" : "TxnLog Replicator", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.Object.wait(Unknown Source)", "app//org.apache.asterix.replication.logging.ReplicationLogBuffer.flush(ReplicationLogBuffer.java:94)", "app//org.apache.asterix.replication.logging.TxnLogReplicator.call(TxnLogReplicator.java:93)", "app//org.apache.asterix.replication.logging.TxnLogReplicator.call(TxnLogReplicator.java:31)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 2, "id" : 54, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2734ad8c", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2131888461 } ], "name" : "ReplicationJobsProcessor", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.replication.management.IndexReplicationManager$ReplicationJobsProcessor.run(IndexReplicationManager.java:174)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 56, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@631b77f7", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2085313771 } ], "name" : "e8a78c35fcb3323712f05201c598bcdeRemoteLogsNotifier", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.replication.logging.RemoteLogsNotifier.run(RemoteLogsNotifier.java:63)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 57, "lock_name" : "java.lang.Object@28bd34a", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 813455850 } ], "name" : "Executor-6:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "app//org.apache.hyracks.storage.common.buffercache.DelayPageCleanerPolicy.notifyCleanCycleFinish(DelayPageCleanerPolicy.java:34)", "app//org.apache.hyracks.storage.common.buffercache.BufferCache$CleanerThread.runCleanCycle(BufferCache.java:773)", "app//org.apache.hyracks.storage.common.buffercache.BufferCache$CleanerThread.run(BufferCache.java:745)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 58, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26c3a732", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 333447190 } ], "name" : "Executor-7:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.asterix.messaging.NCMessageBroker$MessageDeliveryService.run(NCMessageBroker.java:162)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 59, "lock_name" : "com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor@241936ab", "name" : "Thread-8", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "app//com.couchbase.analytics.bootstrap.NsServerHttpClient$IdleConnectionMonitor.run(NsServerHttpClient.java:66)" ], "state" : "TIMED_WAITING" }, { "id" : 60, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1725151361 } ], "name" : "Executor-8:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//org.apache.hyracks.control.common.result.ResultStateSweeper.run(ResultStateSweeper.java:44)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 61, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 470108432, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1078515801, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 2072583509, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9607]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "id" : 62, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 373870817, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 540235505, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 650102974, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "TCPEndpoint IO Thread [/0.0.0.0:9608]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.net.protocols.tcp.TCPEndpoint$IOThread.run(TCPEndpoint.java:133)" ], "state" : "RUNNABLE" }, { "blocked_count" : 1, "id" : 63, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@15ef1a89", "name" : "RMI RenewClean-[127.0.0.1:9614]", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 64, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@f2364f4", "name" : "RMI TCP Connection(idle)", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 65, "lock_name" : "java.util.concurrent.Semaphore$NonfairSync@45700f0d", "name" : "e8a78c35fcb3323712f05201c598bcde-Heartbeat", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.Semaphore.tryAcquire(Unknown Source)", "app//org.apache.hyracks.control.nc.heartbeat.HeartbeatTask.run(HeartbeatTask.java:59)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 24, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f4e5d8c", "name" : "Worker:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.control.common.work.WorkQueue$WorkerThread.run(WorkQueue.java:114)" ], "state" : "WAITING" }, { "id" : 66, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 661422307, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1245375957, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 267881464, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "IPC Network Listener Thread [/0.0.0.0:41713]", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.doRun(IPCConnectionManager.java:191)", "app//org.apache.hyracks.ipc.impl.IPCConnectionManager$NetworkThread.run(IPCConnectionManager.java:181)" ], "state" : "RUNNABLE" }, { "blocked_count" : 71, "id" : 67, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1164891104 } ], "name" : "Executor-9:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.management@10.0.2/sun.management.ThreadImpl.dumpThreads0(Native Method)", "java.management@10.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)", "java.management@10.0.2/sun.management.ThreadImpl.dumpAllThreads(Unknown Source)", "app//org.apache.hyracks.util.ThreadDumpUtil.takeDumpJSON(ThreadDumpUtil.java:57)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$getResultJson$0(NodeDiagnosticsServlet.java:86)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$349/46136157.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 68, "locked_monitors" : [ { "className" : "java.util.Collections$SynchronizedMap", "identityHashCode" : 47711671, "lockedStackDepth" : 0, "lockedStackFrame" : { "classLoaderName" : null, "className" : "java.util.Collections$SynchronizedMap", "fileName" : null, "lineNumber" : -1, "methodName" : "computeIfAbsent", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1922551868 } ], "name" : "Executor-10:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/java.util.Collections$SynchronizedMap.computeIfAbsent(Unknown Source)", "app//org.apache.hyracks.util.StringUtil.toCamelCase(StringUtil.java:36)", "app//org.apache.hyracks.api.config.IOption.camelCase(IOption.java:72)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$populateConfig$4(NodeDiagnosticsServlet.java:164)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$353/913246366.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)", "java.base@10.0.2/java.util.HashMap$KeySpliterator.forEachRemaining(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.copyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.evaluate(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline.forEach(Unknown Source)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.populateConfig(NodeDiagnosticsServlet.java:164)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$getResultJson$1(NodeDiagnosticsServlet.java:87)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$350/95973018.call(Unknown Source)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 69, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@41f0f697", "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1314688755 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileRequestHandler", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ArrayBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileRequestHandler.run(HyracksConnection.java:431)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 70, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1620216417 } ], "name" : "HyracksConnection Uninterrubtible thread: UninterrubtileHandlerWatcher", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "java.base@10.0.2/java.lang.Thread.sleep(Unknown Source)", "java.base@10.0.2/java.util.concurrent.TimeUnit.sleep(Unknown Source)", "app//org.apache.hyracks.api.client.HyracksConnection$UninterrubtileHandlerWatcher.run(HyracksConnection.java:459)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 71, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@5c39ef39", "name" : "process reaper", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 72, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1872893709, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 764756723, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1796689304, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-6-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 55, "locked_monitors" : [ { "className" : "java.lang.Object", "identityHashCode" : 2017736676, "lockedStackDepth" : 2, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.ServerSocketChannelImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "accept", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "e8a78c35fcb3323712f05201c598bcde Replication Channel Thread", "stack" : [ "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)", "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)", "java.base@10.0.2/sun.nio.ch.ServerSocketChannelImpl.accept(Unknown Source)", "app//org.apache.asterix.replication.management.ReplicationChannel.run(ReplicationChannel.java:72)" ], "state" : "RUNNABLE" }, { "blocked_count" : 9, "id" : 73, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 128011678, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 10753347, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1368322882, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 74, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 17, "id" : 75, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@56c74b6", "name" : "Executor-11:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 20, "id" : 76, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25c27755", "name" : "FIFO Writer Thread", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "app//org.apache.hyracks.storage.common.buffercache.AsyncFIFOPageQueueManager.run(AsyncFIFOPageQueueManager.java:163)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 77, "locked_monitors" : [ { "className" : "java.util.Collections$SynchronizedMap", "identityHashCode" : 47711671, "lockedStackDepth" : 0, "lockedStackFrame" : { "classLoaderName" : null, "className" : "java.util.Collections$SynchronizedMap", "fileName" : null, "lineNumber" : -1, "methodName" : "computeIfAbsent", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 1956281320 } ], "name" : "HttpExecutor(port:9611)-1", "stack" : [ "java.base@10.0.2/java.util.Collections$SynchronizedMap.computeIfAbsent(Unknown Source)", "app//org.apache.hyracks.util.StringUtil.toCamelCase(StringUtil.java:36)", "app//org.apache.hyracks.api.config.IOption.camelCase(IOption.java:72)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.lambda$populateConfig$4(NodeDiagnosticsServlet.java:164)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$353/913246366.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline$2$1.accept(Unknown Source)", "java.base@10.0.2/java.util.HashMap$KeySpliterator.forEachRemaining(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.copyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.evaluate(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline.forEach(Unknown Source)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.populateConfig(NodeDiagnosticsServlet.java:164)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.getResultJson(NodeDiagnosticsServlet.java:92)", "app//com.couchbase.analytics.servlet.NodeDiagnosticsServlet.get(NodeDiagnosticsServlet.java:74)", "app//org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:88)", "app//com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:79)", "app//org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:71)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:56)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 78, "name" : "Checkpoint Thread (78)", "stack" : [ "java.base@10.0.2/java.lang.Thread.sleep(Native Method)", "app//org.apache.asterix.transaction.management.service.recovery.CheckpointThread.run(CheckpointThread.java:61)" ], "state" : "TIMED_WAITING" }, { "id" : 79, "name" : "RMI TCP Accept-9613", "stack" : [ "java.base@10.0.2/java.net.PlainSocketImpl.socketAccept(Native Method)", "java.base@10.0.2/java.net.AbstractPlainSocketImpl.accept(Unknown Source)", "java.base@10.0.2/java.net.ServerSocket.implAccept(Unknown Source)", "java.base@10.0.2/java.net.ServerSocket.accept(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 80, "lock_name" : "java.lang.ref.ReferenceQueue$Lock@370944c", "name" : "RMI RenewClean-[127.0.0.1:9613]", "stack" : [ "java.base@10.0.2/java.lang.Object.wait(Native Method)", "java.base@10.0.2/java.lang.ref.ReferenceQueue.remove(Unknown Source)", "java.rmi@10.0.2/sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "id" : 81, "lock_name" : "java.util.concurrent.SynchronousQueue$TransferStack@f2364f4", "name" : "RMI TCP Connection(idle)", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(Unknown Source)", "java.base@10.0.2/java.util.concurrent.SynchronousQueue.poll(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "TIMED_WAITING" }, { "blocked_count" : 1, "id" : 82, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1805128103, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 221712806, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1006221194, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-2", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 83, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 84, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 4, "id" : 85, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1017939533, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 387104086, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 572339251, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-3", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 86, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 87, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-1", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 88, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 89, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 90, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-6", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 3, "id" : 91, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 216590395, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 236068093, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1164250528, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-4", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 92, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 93, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-3", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 94, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-7", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 95, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-8", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 96, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-9", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 97, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-10", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 98, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1711001963, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 773412063, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 812793877, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-5", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 99, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-4", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 100, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-5", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 101, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-11", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 102, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-12", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 103, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-13", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 104, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-14", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 105, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ac0580", "name" : "HttpExecutor(port:9611)-15", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 106, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1570194358, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1725735398, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 34057806, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-6", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 107, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-6", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 108, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-7", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 3, "id" : 109, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 550970611, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1195006380, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1156191036, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-7", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 110, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-8", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 111, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-9", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 112, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 2065739825, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 1649009434, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1999315398, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-7-8", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 113, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-10", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 114, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-11", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 115, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-12", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 116, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-13", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 117, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-14", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 118, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@542e921d", "name" : "HttpExecutor(port:9600)-15", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 119, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 1665974252, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 487799436, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 812604167, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-5-1", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 120, "lock_name" : "java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1ee469dd", "name" : "HttpExecutor(port:9601)-0", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)", "java.base@10.0.2/java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "WAITING" }, { "id" : 121, "locked_monitors" : [ { "className" : "sun.nio.ch.Util$2", "identityHashCode" : 416355890, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "java.util.Collections$UnmodifiableSet", "identityHashCode" : 479365374, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } }, { "className" : "sun.nio.ch.EPollSelectorImpl", "identityHashCode" : 1450135304, "lockedStackDepth" : 3, "lockedStackFrame" : { "classLoaderName" : null, "className" : "sun.nio.ch.SelectorImpl", "fileName" : null, "lineNumber" : -1, "methodName" : "lockAndDoSelect", "moduleName" : "java.base", "moduleVersion" : "10.0.2", "nativeMethod" : false } } ], "name" : "nioEventLoopGroup-5-2", "stack" : [ "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)", "java.base@10.0.2/sun.nio.ch.EPollArrayWrapper.poll(Unknown Source)", "java.base@10.0.2/sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source)", "java.base@10.0.2/sun.nio.ch.SelectorImpl.select(Unknown Source)", "app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)", "app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)", "app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)", "app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 7, "id" : 122, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 284931370 } ], "name" : "HttpExecutor(port:9601)-1", "stack" : [ "java.base@10.0.2/java.net.SocketInputStream.socketRead0(Native Method)", "java.base@10.0.2/java.net.SocketInputStream.socketRead(Unknown Source)", "java.base@10.0.2/java.net.SocketInputStream.read(Unknown Source)", "java.base@10.0.2/java.net.SocketInputStream.read(Unknown Source)", "app//org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)", "app//org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)", "app//org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)", "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)", "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)", "app//org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)", "app//org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)", "app//org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)", "app//org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)", "app//org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)", "app//org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)", "app//org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)", "app//org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)", "app//org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)", "app//org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)", "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)", "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet$DiagnosticRequest.processNode(ClusterDiagnosticsServlet.java:86)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet$DiagnosticRequest$$Lambda$347/1366535344.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)", "java.base@10.0.2/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.copyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)", "java.base@10.0.2/java.util.concurrent.CountedCompleter.exec(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinPool$WorkQueue.sharedHelpCC(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinPool.externalHelpComplete(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinTask.tryExternalHelp(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinTask.externalAwaitDone(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinTask.doInvoke(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinTask.invoke(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.evaluate(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline.forEach(Unknown Source)", "java.base@10.0.2/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet$DiagnosticRequest.execute(ClusterDiagnosticsServlet.java:65)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet.get(ClusterDiagnosticsServlet.java:49)", "app//org.apache.hyracks.http.server.AbstractServlet.handle(AbstractServlet.java:88)", "app//com.couchbase.analytics.servlet.AuthenticatedServlet.handle(AuthenticatedServlet.java:79)", "app//org.apache.hyracks.http.server.HttpRequestHandler.handle(HttpRequestHandler.java:71)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:56)", "app//org.apache.hyracks.http.server.HttpRequestHandler.call(HttpRequestHandler.java:37)", "java.base@10.0.2/java.util.concurrent.FutureTask.run(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "blocked_count" : 1, "id" : 123, "name" : "ForkJoinPool.commonPool-worker-1", "stack" : [ "java.base@10.0.2/java.net.SocketInputStream.socketRead0(Native Method)", "java.base@10.0.2/java.net.SocketInputStream.socketRead(Unknown Source)", "java.base@10.0.2/java.net.SocketInputStream.read(Unknown Source)", "java.base@10.0.2/java.net.SocketInputStream.read(Unknown Source)", "app//org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)", "app//org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)", "app//org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)", "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)", "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)", "app//org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)", "app//org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)", "app//org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)", "app//org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)", "app//org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)", "app//org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)", "app//org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)", "app//org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)", "app//org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)", "app//org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)", "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)", "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet$DiagnosticRequest.processNode(ClusterDiagnosticsServlet.java:86)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet$DiagnosticRequest$$Lambda$347/1366535344.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)", "java.base@10.0.2/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.copyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)", "java.base@10.0.2/java.util.concurrent.CountedCompleter.exec(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 124, "lock_name" : "java.util.concurrent.ForkJoinPool@6b787e99", "name" : "ForkJoinPool.commonPool-worker-2", "stack" : [ "java.base@10.0.2/jdk.internal.misc.Unsafe.park(Native Method)", "java.base@10.0.2/java.util.concurrent.locks.LockSupport.park(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)" ], "state" : "WAITING" }, { "blocked_count" : 3, "id" : 125, "name" : "ForkJoinPool.commonPool-worker-3", "stack" : [ "java.base@10.0.2/java.net.SocketInputStream.socketRead0(Native Method)", "java.base@10.0.2/java.net.SocketInputStream.socketRead(Unknown Source)", "java.base@10.0.2/java.net.SocketInputStream.read(Unknown Source)", "java.base@10.0.2/java.net.SocketInputStream.read(Unknown Source)", "app//org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)", "app//org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)", "app//org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)", "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)", "app//org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)", "app//org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)", "app//org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)", "app//org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)", "app//org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)", "app//org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)", "app//org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)", "app//org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)", "app//org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)", "app//org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)", "app//org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)", "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)", "app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet$DiagnosticRequest.processNode(ClusterDiagnosticsServlet.java:86)", "app//com.couchbase.analytics.servlet.ClusterDiagnosticsServlet$DiagnosticRequest$$Lambda$347/1366535344.accept(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)", "java.base@10.0.2/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)", "java.base@10.0.2/java.util.stream.AbstractPipeline.copyInto(Unknown Source)", "java.base@10.0.2/java.util.stream.ForEachOps$ForEachTask.compute(Unknown Source)", "java.base@10.0.2/java.util.concurrent.CountedCompleter.exec(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)" ], "state" : "RUNNABLE" }, { "id" : 126, "locked_synchronizers" : [ { "className" : "java.util.concurrent.ThreadPoolExecutor$Worker", "identityHashCode" : 2102563797 } ], "name" : "Executor-12:e8a78c35fcb3323712f05201c598bcde", "stack" : [ "java.base@10.0.2/java.lang.Object.hashCode(Native Method)", "java.base@10.0.2/java.util.concurrent.ConcurrentHashMap.putVal(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ConcurrentHashMap.put(Unknown Source)", "app//org.apache.hyracks.util.MaintainedThreadNameExecutorService.beforeExecute(MaintainedThreadNameExecutorService.java:43)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)", "java.base@10.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)", "java.base@10.0.2/java.lang.Thread.run(Unknown Source)" ], "state" : "RUNNABLE" } ] } } ] } [{"name":"beer-sample","bucketType":"membase","authType":"sasl","proxyPort":0,"uri":"/pools/default/buckets/beer-sample?bucket_uuid=bf5db71f1d40b85d094a699c31d73008","streamingUri":"/pools/default/bucketsStreaming/beer-sample?bucket_uuid=bf5db71f1d40b85d094a699c31d73008","localRandomKeyUri":"/pools/default/buckets/beer-sample/localRandomKey","controllers":{"compactAll":"/pools/default/buckets/beer-sample/controller/compactBucket","compactDB":"/pools/default/buckets/beer-sample/controller/compactDatabases","purgeDeletes":"/pools/default/buckets/beer-sample/controller/unsafePurgeBucket","startRecovery":"/pools/default/buckets/beer-sample/controller/startRecovery"},"nodes":[{"couchApiBaseHTTPS":"https://127.0.0.1:19500/beer-sample%2Bbf5db71f1d40b85d094a699c31d73008","couchApiBase":"http://127.0.0.1:9500/beer-sample%2Bbf5db71f1d40b85d094a699c31d73008","systemStats":{"cpu_utilization_rate":79.39698492462311,"swap_total":0,"swap_used":0,"mem_total":16818167808,"mem_free":10094727168},"interestingStats":{"cmd_get":0,"couch_docs_actual_disk_size":42250196,"couch_docs_data_size":39476224,"couch_spatial_data_size":0,"couch_spatial_disk_size":0,"couch_views_actual_disk_size":773677,"couch_views_data_size":769525,"curr_items":9372,"curr_items_tot":9372,"ep_bg_fetched":0,"get_hits":0,"mem_used":26713848,"ops":0,"vb_active_num_non_resident":0,"vb_replica_curr_items":0},"uptime":"142","memoryTotal":16818167808,"memoryFree":10094727168,"mcdMemoryReserved":12831,"mcdMemoryAllocated":12831,"replication":0,"clusterMembership":"active","recoveryType":"none","status":"healthy","otpNode":"n_0@127.0.0.1","thisNode":true,"hostname":"127.0.0.1:9000","clusterCompatibility":393216,"version":"6.0.0-0000-enterprise","os":"x86_64-unknown-linux-gnu","cpuCount":4,"ports":{"httpsMgmt":19000,"httpsCAPI":19500,"proxy":12001,"direct":12000},"services":["cbas","kv"]}],"stats":{"uri":"/pools/default/buckets/beer-sample/stats","directoryURI":"/pools/default/buckets/beer-sample/statsDirectory","nodeStatsListURI":"/pools/default/buckets/beer-sample/nodes"},"nodeLocator":"vbucket","saslPassword":"4e3abf490623b3dd4b2d0faf9f8d88b9","ddocs":{"uri":"/pools/default/buckets/beer-sample/ddocs"},"replicaIndex":true,"autoCompactionSettings":false,"uuid":"bf5db71f1d40b85d094a699c31d73008","maxTTL":0,"compressionMode":"passive","replicaNumber":1,"threadsNumber":3,"quota":{"ram":104857600,"rawRAM":104857600},"basicStats":{"storageTotals":{"ram":{"total":16818167808,"quotaTotal":3221225472,"quotaUsed":104857600,"used":14395076608,"usedByData":26713848,"quotaUsedPerNode":104857600,"quotaTotalPerNode":3221225472},"hdd":{"total":137813643264,"quotaTotal":137813643264,"used":38587820113,"usedByData":43023873,"free":99225823151}},"quotaPercentUsed":25.47631072998047,"opsPerSec":0,"diskFetches":0,"itemCount":9372,"diskUsed":43023873,"dataUsed":40245749,"memUsed":26713848,"vbActiveNumNonResident":0},"evictionPolicy":"valueOnly","conflictResolutionType":"seqno","bucketCapabilitiesVer":"","bucketCapabilities":["couchapi","xattr","dcp","cbhello","touch","cccp","xdcrCheckpointing","nodesExt"]}] 2018-08-12T14:33:30.045-07:00 INFO TestExecutor [main] Cleaning up after test case recovery: kill-cc-multiple-nodes... 2018-08-12T14:33:30.197-07:00 INFO TestExecutor [main] Clean up completed for test case recovery: kill-cc-multiple-nodes 2018-08-12T14:33:30.198-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:33:30.202-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:33:30.838-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:33:30.845-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:33:30.846-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:33:32.857-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 19: recovery: kill-cc-multiple-nodes]'... 2018-08-12T14:33:33.147-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 19: recovery: kill-cc-multiple-nodes] FAILED (java.lang.Exception) 2018-08-12T14:33:33.152-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 20: recovery: kill-nc-then-partial-rollback] START 2018-08-12T14:33:33.261-07:00 INFO TestExecutor [main] Starting [TEST]: recovery/kill-nc-then-partial-rollback ... 2018-08-12T14:33:33.262-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:33:45.412-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:33:45.449-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:33:45.449-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:33:45.450-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/analytics_datadir/target/iodevicestest/iodevices/1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/analytics_datadir/target/iodevicestest/iodevices/2] 2018-08-12T14:33:51.126-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:33:51.130-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:33:51.453-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:33:51.460-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:33:51.460-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-29-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:34:07.056-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:34:07.061-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) does not exist. 2018-08-12T14:34:07.070-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) created successfully. 2018-08-12T14:34:11.635-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json 2018-08-12T14:34:13.894-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p1/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:13.901-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:13.902-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json 2018-08-12T14:34:16.473-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p2/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:16.476-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:16.476-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json 2018-08-12T14:34:19.037-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p3/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:19.040-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:19.040-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json 2018-08-12T14:34:20.914-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p4/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:20.920-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:20.920-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p5/gbook_users.json 2018-08-12T14:34:22.738-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p5/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:22.745-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:22.746-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p6/gbook_users.json 2018-08-12T14:34:25.448-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p6/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:25.452-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:25.453-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p7/gbook_users.json 2018-08-12T14:34:28.508-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p7/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:28.511-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:28.512-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p8/gbook_users.json 2018-08-12T14:34:30.743-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p8/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:30.746-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:30.746-07:00 INFO CBImportCommand [main] Executing: sh -c ./cbimport json -c 127.0.0.1:9000 -u couchbase -p couchbase -b gbook_users -g %id% -f lines -d file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p9/gbook_users.json 2018-08-12T14:34:33.128-07:00 INFO ClusterExecutionITBase [main+] >> Json `file:///home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/ingestion/src/test/resources/data/p9/gbook_users.json` imported to `http://127.0.0.1:9000` successfully 2018-08-12T14:34:33.131-07:00 INFO TestExecutor [main] cbimport exit code 0 2018-08-12T14:34:33.172-07:00 INFO TestExecutor [main] Found Write queue of bucket gbook_users to be 1651 2018-08-12T14:34:34.194-07:00 INFO TestExecutor [main] Bucket gbook_users has been persisted 2018-08-12T14:34:34.219-07:00 INFO CBStopPersistenceCommand [main] Executing: sh -c ./cbepctl 127.0.0.1:12000 stop -u couchbase -p couchbase -b gbook_users 2018-08-12T14:34:35.065-07:00 INFO ClusterExecutionITBase [main+] >> Persistence stopped 2018-08-12T14:34:35.073-07:00 INFO TestExecutor [main] CBStopPersistenceCommand exit code: 0 2018-08-12T14:34:36.447-07:00 INFO TestExecutor [pool-449-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.009.adm 2018-08-12T14:34:36.450-07:00 INFO TestExecutor [pool-449-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.009.adm 2018-08-12T14:34:36.450-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.009.pollquery.sqlpp changed at line 1: expected < { "count": 90004 } actual > { "count": 87474 } 2018-08-12T14:34:37.826-07:00 INFO TestExecutor [pool-449-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.009.adm 2018-08-12T14:34:39.626-07:00 INFO TestExecutor [main] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.011.adm 2018-08-12T14:34:39.628-07:00 INFO KillAllCommand [main] Executing: sh -c pkill -9 memcached 2018-08-12T14:34:39.661-07:00 INFO TestExecutor [main] KillAllCommand exit code: 0 2018-08-12T14:34:39.663-07:00 INFO ClusterExecutionITBase [main] Killing node 0 (signalNum: 9) 2018-08-12T14:34:40.474-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:34:40.571-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:34:40.599-07:00 ERRO asterix [pool-455-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:40.602-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:41.615-07:00 ERRO asterix [pool-456-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:41.617-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:42.633-07:00 ERRO asterix [pool-457-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:42.636-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:43.648-07:00 ERRO asterix [pool-458-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:43.649-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:44.653-07:00 ERRO asterix [pool-459-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:44.655-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:45.676-07:00 ERRO asterix [pool-460-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:45.678-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:46.682-07:00 ERRO asterix [pool-461-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:46.683-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:47.697-07:00 ERRO asterix [pool-462-thread-1] Failure executing POST http://localhost:9600/analytics/service HTTP/1.1 org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.asterix.test.common.TestExecutor.lambda$executeHttpRequest$2(TestExecutor.java:500) ~[asterix-app-0.9.4-SNAPSHOT-tests.jar:6.0.0-0000] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_131] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_131] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_131] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_131] at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2] at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2] ... 13 more 2018-08-12T14:34:47.701-07:00 INFO TestExecutor [main] Received exception on poll: java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:9600 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused) 2018-08-12T14:34:49.401-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Analytics Service is temporarily unavailable 2018-08-12T14:34:52.590-07:00 INFO TestExecutor [pool-454-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.014.adm 2018-08-12T14:34:56.208-07:00 INFO TestExecutor [pool-466-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:34:56.209-07:00 INFO TestExecutor [pool-466-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:34:56.209-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 56760 } 2018-08-12T14:34:58.520-07:00 INFO TestExecutor [pool-466-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:34:58.521-07:00 INFO TestExecutor [pool-466-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:34:58.521-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 58500 } 2018-08-12T14:35:00.226-07:00 INFO TestExecutor [pool-466-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:35:00.227-07:00 INFO TestExecutor [pool-466-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:35:00.227-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 66905 } 2018-08-12T14:35:01.845-07:00 INFO TestExecutor [pool-466-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:35:01.845-07:00 INFO TestExecutor [pool-466-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:35:01.846-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 77193 } 2018-08-12T14:35:03.391-07:00 INFO TestExecutor [pool-466-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:35:03.392-07:00 INFO TestExecutor [pool-466-thread-1] Actual results file: target/rttest/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:35:03.392-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.pollquery.sqlpp changed at line 1: expected < { "count": 90000 } actual > { "count": 87524 } 2018-08-12T14:35:04.675-07:00 INFO TestExecutor [pool-466-thread-1] Expected results file: src/test/resources/runtimets/results/recovery/kill-nc-then-partial-rollback/kill-nc-then-partial-rollback.016.adm 2018-08-12T14:35:06.376-07:00 INFO TestExecutor [main] [TEST]: recovery/kill-nc-then-partial-rollback PASSED 2018-08-12T14:35:06.376-07:00 INFO TestExecutor [main] Cleaning up after test case recovery: kill-nc-then-partial-rollback... 2018-08-12T14:35:06.437-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:35:06.471-07:00 INFO TestExecutor [main] Disconnecting link Metadata.Local 2018-08-12T14:35:07.526-07:00 INFO KvStoreHttpHelper [main] Bucket (gbook_users) deleted successfully. 2018-08-12T14:35:07.527-07:00 INFO TestExecutor [main] Clean up completed for test case recovery: kill-nc-then-partial-rollback 2018-08-12T14:35:07.527-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:35:07.530-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:35:08.314-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:35:08.323-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:35:08.323-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:35:10.329-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 20: recovery: kill-nc-then-partial-rollback]'... 2018-08-12T14:35:10.377-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 20: recovery: kill-nc-then-partial-rollback] SUCCEEDED 2018-08-12T14:35:10.378-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 21: replication: adjust_replicas_on_rebalance] START 2018-08-12T14:35:10.403-07:00 INFO TestExecutor [main] Starting [TEST]: replication/adjust_replicas_on_rebalance ... 2018-08-12T14:35:10.404-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:35:23.305-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:35:23.364-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:35:23.364-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:35:23.365-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:35:28.704-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:35:28.707-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:35:28.998-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:35:29.001-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:35:29.002-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-30-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:35:45.039-07:00 INFO TestExecutor [pool-481-thread-1] Expected results file: src/test/resources/runtimets/results/replication/adjust_replicas_on_rebalance/adjust_replicas_on_rebalance.1.adm 2018-08-12T14:35:45.040-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1, 2, 3] 2018-08-12T14:35:45.063-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1, 2, 3]]) are fully removed 2018-08-12T14:35:45.073-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:35:45.073-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1, 2, 3] with service(s): [ANALYTICS] 2018-08-12T14:35:49.813-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:35:49.864-07:00 INFO ClusterExecutionITBase [main -> init n_3+] >> SUCCESS: Node initialized 2018-08-12T14:35:50.112-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:35:50.114-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9001,127.0.0.1:9003,127.0.0.1:9002, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:36:03.504-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:36:03.509-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:36:25.008-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:36:25.009-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:36:25.020-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2, 3], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS},{ANALYTICS}}, cbas node count: 4 2018-08-12T14:36:25.020-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-31-/127.0.0.1:9630] Waiting for NCs == 4 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:25.020-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-31-/127.0.0.1:9645] Waiting for NCs == 4 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:25.020-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-31-/127.0.0.1:9615] Waiting for NCs == 4 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:25.020-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-31-/127.0.0.1:9600] Waiting for NCs == 4 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:25.413-07:00 INFO TestExecutor [pool-483-thread-1] Expected results file: src/test/resources/runtimets/results/replication/adjust_replicas_on_rebalance/adjust_replicas_on_rebalance.3.regex 2018-08-12T14:36:25.415-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0, 1, 2, 3] to [0, 1] 2018-08-12T14:36:25.423-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([]) are fully removed 2018-08-12T14:36:25.423-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:36:25.423-07:00 INFO ClusterExecutionITBase [main] removing nodes [2, 3] 2018-08-12T14:36:25.423-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-remove, 127.0.0.1:9002,127.0.0.1:9003] 2018-08-12T14:36:28.777-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:36:28.777-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:36:28.784-07:00 INFO ClusterExecutionITBase [main] state: [0, 1], nodes: {{DATA,ANALYTICS},{ANALYTICS},{},{}}, cbas node count: 2 2018-08-12T14:36:28.784-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-32-/127.0.0.1:9615] Waiting for NCs == 2 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:28.784-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-32-/127.0.0.1:9600] Waiting for NCs == 2 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:28.888-07:00 INFO TestExecutor [pool-485-thread-1] Expected results file: src/test/resources/runtimets/results/replication/adjust_replicas_on_rebalance/adjust_replicas_on_rebalance.5.regex 2018-08-12T14:36:28.890-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0, 1] to [0, 1, 2] 2018-08-12T14:36:28.896-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[2]]) are fully removed 2018-08-12T14:36:33.910-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:36:33.911-07:00 INFO ClusterExecutionITBase [main] adding node(s) [2] with service(s): [ANALYTICS] 2018-08-12T14:36:34.273-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:36:34.291-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9002, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:36:40.431-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:36:40.434-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:36:52.789-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:36:52.789-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:36:52.793-07:00 INFO ClusterExecutionITBase [main] state: [0, 1, 2], nodes: {{DATA,ANALYTICS},{ANALYTICS},{ANALYTICS},{}}, cbas node count: 3 2018-08-12T14:36:52.793-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-33-/127.0.0.1:9615] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:52.793-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-33-/127.0.0.1:9600] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:52.793-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-33-/127.0.0.1:9630] Waiting for NCs == 3 & Cluster ACTIVE for up to 60s... 2018-08-12T14:36:53.359-07:00 INFO TestExecutor [pool-487-thread-1] Expected results file: src/test/resources/runtimets/results/replication/adjust_replicas_on_rebalance/adjust_replicas_on_rebalance.7.regex 2018-08-12T14:36:53.360-07:00 INFO TestExecutor [main] [TEST]: replication/adjust_replicas_on_rebalance PASSED 2018-08-12T14:36:53.361-07:00 INFO TestExecutor [main] Cleaning up after test case replication: adjust_replicas_on_rebalance... 2018-08-12T14:36:54.890-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:36:56.387-07:00 INFO TestExecutor [main] Clean up completed for test case replication: adjust_replicas_on_rebalance 2018-08-12T14:36:56.387-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:36:56.439-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:36:57.286-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:36:57.293-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:36:57.293-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:36:59.299-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 21: replication: adjust_replicas_on_rebalance]'... 2018-08-12T14:36:59.389-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 21: replication: adjust_replicas_on_rebalance] SUCCEEDED 2018-08-12T14:36:59.390-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 22: replication: survive_node_restart] START 2018-08-12T14:36:59.444-07:00 INFO TestExecutor [main] Starting [TEST]: replication/survive_node_restart ... 2018-08-12T14:36:59.445-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:37:09.478-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:37:09.493-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:37:09.493-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:37:09.493-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:37:13.564-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:37:13.568-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:37:13.836-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:37:13.841-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{},{}}, cbas node count: 1 2018-08-12T14:37:13.841-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-34-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:37:31.036-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1] 2018-08-12T14:37:31.059-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1]]) are fully removed 2018-08-12T14:37:31.063-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:37:31.063-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1] with service(s): [ANALYTICS] 2018-08-12T14:37:35.215-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:37:35.220-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:37:39.513-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:37:39.518-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:37:53.826-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:37:53.827-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:37:53.831-07:00 INFO ClusterExecutionITBase [main] state: [0, 1], nodes: {{DATA,ANALYTICS},{ANALYTICS},{},{}}, cbas node count: 2 2018-08-12T14:37:53.831-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-35-/127.0.0.1:9615] Waiting for NCs == 2 & Cluster ACTIVE for up to 60s... 2018-08-12T14:37:53.832-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-35-/127.0.0.1:9600] Waiting for NCs == 2 & Cluster ACTIVE for up to 60s... 2018-08-12T14:37:54.467-07:00 INFO TestExecutor [pool-494-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.1.regex 2018-08-12T14:37:54.469-07:00 INFO ClusterExecutionITBase [main] Killing node 0 (signalNum: 15) 2018-08-12T14:37:54.958-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:37:57.985-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:37:57.986-07:00 INFO TestExecutor [main] Waiting for cluster state ACTIVE... 2018-08-12T14:38:09.469-07:00 INFO TestExecutor [main] Cluster state now ACTIVE 2018-08-12T14:38:09.509-07:00 INFO TestExecutor [pool-496-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.4.regex 2018-08-12T14:38:09.510-07:00 INFO ClusterExecutionITBase [main] Killing node 1 (signalNum: 15) 2018-08-12T14:38:10.098-07:00 INFO ClusterExecutionITBase [main] Waiting for killed process to exit ... 2018-08-12T14:38:12.948-07:00 INFO ClusterExecutionITBase [main] Killed process has exited! 2018-08-12T14:38:12.949-07:00 INFO TestExecutor [main] Waiting for cluster state ACTIVE... 2018-08-12T14:38:13.018-07:00 INFO TestExecutor [main] Cluster state now ACTIVE 2018-08-12T14:38:13.067-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:13.068-07:00 INFO TestExecutor [pool-498-thread-1] Actual results file: target/rttest/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:13.069-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Result for src/test/resources/runtimets/queries/replication/survive_node_restart/survive_node_restart.7.pollget.http: expected pattern ' "status" : "IN_SYNC"' not found in result: [ { "partition" : 0, "replicas" : [ { "location" : "127.0.0.1:9627", "status" : "DISCONNECTED" } ] } ] 2018-08-12T14:38:14.098-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:14.099-07:00 INFO TestExecutor [pool-498-thread-1] Actual results file: target/rttest/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:14.099-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Result for src/test/resources/runtimets/queries/replication/survive_node_restart/survive_node_restart.7.pollget.http: expected pattern ' "status" : "IN_SYNC"' not found in result: [ { "partition" : 0, "replicas" : [ { "location" : "127.0.0.1:9627", "status" : "DISCONNECTED" } ] } ] 2018-08-12T14:38:15.131-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:15.131-07:00 INFO TestExecutor [pool-498-thread-1] Actual results file: target/rttest/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:15.131-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Result for src/test/resources/runtimets/queries/replication/survive_node_restart/survive_node_restart.7.pollget.http: expected pattern ' "status" : "IN_SYNC"' not found in result: [ { "partition" : 0, "replicas" : [ { "location" : "127.0.0.1:9627", "status" : "DISCONNECTED" } ] } ] 2018-08-12T14:38:16.171-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:16.171-07:00 INFO TestExecutor [pool-498-thread-1] Actual results file: target/rttest/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:16.172-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Result for src/test/resources/runtimets/queries/replication/survive_node_restart/survive_node_restart.7.pollget.http: expected pattern ' "status" : "IN_SYNC"' not found in result: [ { "partition" : 0, "replicas" : [ { "location" : "127.0.0.1:9627", "status" : "DISCONNECTED" } ] } ] 2018-08-12T14:38:17.214-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:17.215-07:00 INFO TestExecutor [pool-498-thread-1] Actual results file: target/rttest/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:17.215-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Result for src/test/resources/runtimets/queries/replication/survive_node_restart/survive_node_restart.7.pollget.http: expected pattern ' "status" : "IN_SYNC"' not found in result: [ { "partition" : 0, "replicas" : [ { "location" : "127.0.0.1:9627", "status" : "DISCONNECTED" } ] } ] 2018-08-12T14:38:18.240-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:18.241-07:00 INFO TestExecutor [pool-498-thread-1] Actual results file: target/rttest/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:18.241-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Result for src/test/resources/runtimets/queries/replication/survive_node_restart/survive_node_restart.7.pollget.http: expected pattern ' "status" : "IN_SYNC"' not found in result: [ { "partition" : 0, "replicas" : [ { "location" : "127.0.0.1:9627", "status" : "DISCONNECTED" } ] } ] 2018-08-12T14:38:19.273-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:19.274-07:00 INFO TestExecutor [pool-498-thread-1] Actual results file: target/rttest/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:19.274-07:00 INFO TestExecutor [main] Received exception on poll: java.lang.Exception: Result for src/test/resources/runtimets/queries/replication/survive_node_restart/survive_node_restart.7.pollget.http: expected pattern ' "status" : "IN_SYNC"' not found in result: [ { "partition" : 0, "replicas" : [ { "location" : "127.0.0.1:9627", "status" : "DISCONNECTED" } ] } ] 2018-08-12T14:38:20.292-07:00 INFO TestExecutor [pool-498-thread-1] Expected results file: src/test/resources/runtimets/results/replication/survive_node_restart/survive_node_restart.7.regex 2018-08-12T14:38:20.293-07:00 INFO TestExecutor [main] [TEST]: replication/survive_node_restart PASSED 2018-08-12T14:38:20.293-07:00 INFO TestExecutor [main] Cleaning up after test case replication: survive_node_restart... 2018-08-12T14:38:22.376-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:38:22.470-07:00 INFO TestExecutor [main] Disconnecting link Metadata.Local 2018-08-12T14:38:23.160-07:00 INFO TestExecutor [main] Clean up completed for test case replication: survive_node_restart 2018-08-12T14:38:23.160-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:38:23.185-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:38:23.772-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:38:23.779-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:38:23.779-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:38:25.784-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 22: replication: survive_node_restart]'... 2018-08-12T14:38:25.855-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 22: replication: survive_node_restart] SUCCEEDED 2018-08-12T14:38:25.856-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 23: replication: ensure_replicas_on_cc_rebalance_out] START 2018-08-12T14:38:25.891-07:00 INFO TestExecutor [main] Starting [TEST]: replication/ensure_replicas_on_cc_rebalance_out ... 2018-08-12T14:38:25.892-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:38:38.515-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:38:38.549-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:38:38.550-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA] 2018-08-12T14:38:38.550-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:38:43.472-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:38:43.475-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:38:43.843-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:38:43.847-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA},{},{},{}}, cbas node count: 0 2018-08-12T14:38:43.855-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0] to [0, 1] 2018-08-12T14:38:43.939-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[1]]) are fully removed 2018-08-12T14:38:43.942-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:38:43.942-07:00 INFO ClusterExecutionITBase [main] adding node(s) [1] with service(s): [ANALYTICS] 2018-08-12T14:38:49.077-07:00 INFO ClusterExecutionITBase [main -> init n_1+] >> SUCCESS: Node initialized 2018-08-12T14:38:49.080-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9001, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:38:54.520-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:38:54.524-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase] 2018-08-12T14:38:58.776-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:38:58.776-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:38:58.779-07:00 INFO ClusterExecutionITBase [main] state: [0, 1], nodes: {{DATA},{ANALYTICS},{},{}}, cbas node count: 1 2018-08-12T14:38:58.780-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-37-/127.0.0.1:9615] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:39:07.650-07:00 INFO CBDocLoaderCommand [main] Executing: sh -c ./bin/cbdocloader -c 127.0.0.1:9000 -u couchbase -p couchbase -b beer-sample -m 100 -d ./samples/beer-sample.zip 2018-08-12T14:39:16.738-07:00 INFO ClusterExecutionITBase [main+] >> Errors occurred during the index creation phase. See logs for details. 2018-08-12T14:39:16.742-07:00 INFO TestExecutor [main] cbdocloader exit code 1 2018-08-12T14:39:22.985-07:00 INFO TestExecutor [pool-515-thread-1] Expected results file: src/test/resources/runtimets/results/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.3.adm 2018-08-12T14:39:22.997-07:00 INFO TestExecutor [pool-515-thread-1] Actual results file: target/rttest/results/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.3.adm 2018-08-12T14:39:22.997-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.3.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 5 } 2018-08-12T14:39:24.203-07:00 INFO TestExecutor [pool-515-thread-1] Expected results file: src/test/resources/runtimets/results/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.3.adm 2018-08-12T14:39:24.203-07:00 INFO TestExecutor [pool-515-thread-1] Actual results file: target/rttest/results/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.3.adm 2018-08-12T14:39:24.210-07:00 INFO TestExecutor [main] Comparison failure on poll: Result for src/test/resources/runtimets/queries/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.3.pollquery.sqlpp changed at line 1: expected < { "count": 5891 } actual > { "count": 4377 } 2018-08-12T14:39:25.399-07:00 INFO TestExecutor [pool-515-thread-1] Expected results file: src/test/resources/runtimets/results/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.3.adm 2018-08-12T14:39:25.406-07:00 INFO ClusterExecutionITBase [main] Rebalancing from [0, 1] to [0, 2] 2018-08-12T14:39:25.418-07:00 INFO ClusterExecutionITBase [main] ensuring joining nodes ([[2]]) are fully removed 2018-08-12T14:39:25.430-07:00 INFO ClusterExecutionITBase [main] ensured joining nodes are fully removed (failures: []) 2018-08-12T14:39:25.430-07:00 INFO ClusterExecutionITBase [main] adding node(s) [2] with service(s): [ANALYTICS] 2018-08-12T14:39:31.384-07:00 INFO ClusterExecutionITBase [main -> init n_2+] >> SUCCESS: Node initialized 2018-08-12T14:39:31.388-07:00 INFO ClusterExecutionITBase [main] Running cli: [server-add, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-add, 127.0.0.1:9002, --server-add-username, couchbase, --server-add-password, couchbase, --services, analytics] 2018-08-12T14:39:36.293-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Server added 2018-08-12T14:39:36.297-07:00 INFO ClusterExecutionITBase [main] removing nodes [1] 2018-08-12T14:39:36.298-07:00 INFO ClusterExecutionITBase [main] calling rebalance with following args: [rebalance, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --server-remove, 127.0.0.1:9001] 2018-08-12T14:39:51.665-07:00 INFO ClusterExecutionITBase [main+] >> Unable to display progress bar on this os 2018-08-12T14:39:51.666-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Rebalance complete 2018-08-12T14:39:51.668-07:00 INFO ClusterExecutionITBase [main] state: [0, 2], nodes: {{DATA},{},{ANALYTICS},{}}, cbas node count: 1 2018-08-12T14:39:51.669-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-38-/127.0.0.1:9630] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:40:06.868-07:00 INFO TestExecutor [pool-519-thread-1] Expected results file: src/test/resources/runtimets/results/replication/ensure_replicas_on_cc_rebalance_out/ensure_replicas_on_cc_rebalance_out.5.adm 2018-08-12T14:40:06.869-07:00 INFO TestExecutor [main] [TEST]: replication/ensure_replicas_on_cc_rebalance_out PASSED 2018-08-12T14:40:06.869-07:00 INFO TestExecutor [main] Cleaning up after test case replication: ensure_replicas_on_cc_rebalance_out... 2018-08-12T14:40:06.985-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:40:10.104-07:00 INFO TestExecutor [main] Disconnecting link Metadata.Local 2018-08-12T14:40:10.278-07:00 INFO TestExecutor [main] Dropping dataset beers 2018-08-12T14:40:10.747-07:00 INFO TestExecutor [main] Clean up completed for test case replication: ensure_replicas_on_cc_rebalance_out 2018-08-12T14:40:10.747-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:40:10.750-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:40:11.344-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:40:11.351-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:40:11.352-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:40:13.356-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 23: replication: ensure_replicas_on_cc_rebalance_out]'... 2018-08-12T14:40:13.607-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 23: replication: ensure_replicas_on_cc_rebalance_out] SUCCEEDED 2018-08-12T14:40:13.609-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 24: unsupported: create_function] START 2018-08-12T14:40:13.721-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:40:26.241-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:40:26.259-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:40:26.259-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:40:26.259-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:40:31.927-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:40:31.931-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:40:32.250-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:40:32.253-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{},{}}, cbas node count: 1 2018-08-12T14:40:32.253-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-39-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:40:50.751-07:00 INFO TestExecutor [main] Starting [TEST]: unsupported/create_function ... 2018-08-12T14:40:51.278-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/unsupported/create_function/create_function.01.ddl.sqlpp raised an (expected) exception 2018-08-12T14:40:51.278-07:00 INFO TestExecutor [main] [TEST]: unsupported/create_function PASSED 2018-08-12T14:40:51.278-07:00 INFO TestExecutor [main] Cleaning up after test case unsupported: create_function... 2018-08-12T14:40:52.012-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:40:52.073-07:00 INFO TestExecutor [main] Disconnecting link testdv.Local 2018-08-12T14:40:52.527-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: testdv 2018-08-12T14:40:52.844-07:00 INFO TestExecutor [main] Clean up completed for test case unsupported: create_function 2018-08-12T14:40:52.845-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:40:52.852-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:40:53.463-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:40:53.470-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:40:53.470-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:40:55.476-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 24: unsupported: create_function]'... 2018-08-12T14:40:55.528-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 24: unsupported: create_function] SUCCEEDED 2018-08-12T14:40:55.529-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 25: unsupported: write-output] START 2018-08-12T14:40:55.582-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:41:06.723-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:41:06.738-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:41:06.738-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:41:06.738-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:41:12.573-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:41:12.576-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:41:12.867-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:41:12.871-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{},{}}, cbas node count: 1 2018-08-12T14:41:12.872-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-40-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:41:28.906-07:00 INFO TestExecutor [main] Starting [TEST]: unsupported/write-output ... 2018-08-12T14:41:29.155-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/unsupported/write-output/write-output.01.query.sqlpp raised an (expected) exception 2018-08-12T14:41:29.155-07:00 INFO TestExecutor [main] [TEST]: unsupported/write-output PASSED 2018-08-12T14:41:29.156-07:00 INFO TestExecutor [main] Cleaning up after test case unsupported: write-output... 2018-08-12T14:41:30.110-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:41:30.613-07:00 INFO TestExecutor [main] Clean up completed for test case unsupported: write-output 2018-08-12T14:41:30.613-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:41:30.622-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:41:31.243-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:41:31.250-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:41:31.250-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:41:33.257-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 25: unsupported: write-output]'... 2018-08-12T14:41:33.305-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 25: unsupported: write-output] SUCCEEDED 2018-08-12T14:41:33.306-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 26: unsupported: multiple-statements] START 2018-08-12T14:41:33.348-07:00 INFO ClusterExecutionITBase [main] starting cluster_run 2018-08-12T14:41:44.388-07:00 INFO ClusterExecutionITBase [waitForCondition] +/node/controller/rename (n_0) 2018-08-12T14:41:44.396-07:00 INFO ClusterExecutionITBase [main] cluster is running 2018-08-12T14:41:44.396-07:00 INFO ClusterExecutionITBase [main] configuring cluster with services: [DATA, ANALYTICS] 2018-08-12T14:41:44.396-07:00 INFO ClusterExecutionITBase [main] Running cli: [node-init, -c, 127.0.0.1:9000, -u, couchbase, -p, couchbase, --node-init-data-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-index-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir, --node-init-hostname, 127.0.0.1, --node-init-analytics-path, /home/couchbase/jenkins/workspace/cbas-jre10-test/ns_server/data/n_0/datadir] 2018-08-12T14:41:49.509-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Node initialized 2018-08-12T14:41:49.513-07:00 INFO ClusterExecutionITBase [main] Running cli: [cluster-init, --cluster-username, couchbase, --cluster-password, couchbase, -c, localhost:9000, --services, data,analytics, --cluster-analytics-ramsize, 3072, --cluster-ramsize, 3072] 2018-08-12T14:41:49.855-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Cluster initialized 2018-08-12T14:41:49.860-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS},{},{},{}}, cbas node count: 1 2018-08-12T14:41:49.860-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-41-/127.0.0.1:9600] Waiting for NCs == 1 & Cluster ACTIVE for up to 60s... 2018-08-12T14:42:05.073-07:00 INFO TestExecutor [main] Starting [TEST]: unsupported/multiple-statements ... 2018-08-12T14:42:05.243-07:00 INFO TestExecutor [main] testFile src/test/resources/runtimets/queries/unsupported/multiple-statements/multiple-statements.01.query.sqlpp raised an (expected) exception 2018-08-12T14:42:05.243-07:00 INFO TestExecutor [main] [TEST]: unsupported/multiple-statements PASSED 2018-08-12T14:42:05.243-07:00 INFO TestExecutor [main] Cleaning up after test case unsupported: multiple-statements... 2018-08-12T14:42:05.960-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2018-08-12T14:42:06.351-07:00 INFO TestExecutor [main] Clean up completed for test case unsupported: multiple-statements 2018-08-12T14:42:06.351-07:00 INFO ClusterExecutionITBase [main] shutting down cluster_run... 2018-08-12T14:42:06.365-07:00 INFO ClusterExecutionITBase [main] cluster_run sent SIGINT 2018-08-12T14:42:06.946-07:00 INFO ClusterExecutionITBase [main] cluster_run has exited 2018-08-12T14:42:06.954-07:00 INFO ClusterExecutionITBase [main] Waiting for 'CB cluster (incl. analytics nodes) shutdown' condition for up to 30 seconds... 2018-08-12T14:42:06.955-07:00 INFO ClusterExecutionITBase [waitForCondition] Ensuring the following ports are not active: [9000, 9001, 9002, 9003, 9004, 9495, 9496, 9497, 9498, 9499, 9500, 9501, 9502, 9503, 9504, 9600, 9601, 9609, 9610, 9611, 9615, 9616, 9624, 9625, 9626, 9630, 9631, 9639, 9640, 9641, 9645, 9646, 9654, 9655, 9656, 9660, 9661, 9669, 9670, 9671, 10000, 10001, 10002, 10003, 10004, 11983, 11987, 11991, 11995, 11999, 12000, 12001, 12002, 12003, 12004, 12005, 12006, 12007, 12008, 12009] 2018-08-12T14:42:08.962-07:00 INFO ClusterExecutionITJava10 [main] Copying logs+data to 'target/ClusterExecutionITJava10/test[ClusterExecutionITJava10 26: unsupported: multiple-statements]'... 2018-08-12T14:42:09.009-07:00 WARN TestMethodTracer [main] ### test[ClusterExecutionITJava10 26: unsupported: multiple-statements] SUCCEEDED [ERROR] Tests run: 27, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,994.137 s <<< FAILURE! - in com.couchbase.analytics.test.cbserver.ClusterExecutionITJava10 [ERROR] test[ClusterExecutionITJava10 19: recovery: kill-cc-multiple-nodes](com.couchbase.analytics.test.cbserver.ClusterExecutionITJava10) Time elapsed: 151.182 s <<< ERROR! java.lang.Exception: Test "src/test/resources/runtimets/queries/recovery/kill-cc-multiple-nodes/kill-cc-multiple-nodes.012.polldynamic.sqlpp" FAILED! at com.couchbase.analytics.test.cbserver.ClusterExecutionITJava10.test(ClusterExecutionITJava10.java:42) Caused by: java.lang.Exception: Poll limit (60s) exceeded without obtaining expected result at com.couchbase.analytics.test.cbserver.ClusterExecutionITJava10.test(ClusterExecutionITJava10.java:42) [INFO] [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ClusterExecutionITJava10.test:42->ClusterExecutionITBase.runTest:106 ยป Test "... [INFO] [ERROR] Tests run: 27, Failures: 0, Errors: 1, Skipped: 0 [INFO] [JENKINS] Recording test results [WARNING] Attempt to (de-)serialize anonymous class hudson.maven.reporters.SurefireArchiver$2; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:verify (default) @ cbas-cbserver-test --- [ERROR] There are test failures. Please refer to /home/couchbase/jenkins/workspace/cbas-jre10-test/analytics/cbas/cbas-test/cbas-cbserver-test/target/failsafe-reports for the individual test results. Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream. [JENKINS] Recording test results [WARNING] Attempt to (de-)serialize anonymous class hudson.maven.reporters.MavenArtifactArchiver$2; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ [JENKINS] Archiving disabled [WARNING] Attempt to (de-)serialize anonymous class hudson.maven.reporters.MavenFingerprinter$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/