[INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building cbas-docker-test 7.0.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-antrun-plugin:1.8:run (check-pom-packaging) @ cbas-docker-test --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (ensure-no-tests-for-pom-packaging) @ cbas-docker-test --- [INFO] Skipping Rule Enforcement. [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (enforce-versions) @ cbas-docker-test --- [INFO] [INFO] --- maven-enforcer-plugin:3.0.0-M3:enforce (ensure-no-duplicate-deps) @ cbas-docker-test --- [INFO] [INFO] --- maven-checkstyle-plugin:3.0.0:check (verify-style) @ cbas-docker-test --- [INFO] [INFO] --- formatter-maven-plugin:2.0.1:format (default) @ cbas-docker-test --- [INFO] Formatting is skipped [INFO] [INFO] --- impsort-maven-plugin:1.2.0:sort (default) @ cbas-docker-test --- [INFO] Total Files Processed: 37 in 00:00.129 [INFO] Already Sorted: 37 [INFO] Needed Sorting: 0 [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (process-resource-bundles) @ cbas-docker-test --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ cbas-docker-test --- [INFO] [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ cbas-docker-test --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ cbas-docker-test --- [INFO] No sources to compile [INFO] [INFO] --- maven-antrun-plugin:1.8:run (default) @ cbas-docker-test --- [INFO] Executing tasks main: [echo] Creating docker-image.properties file [mkdir] Created dir: /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/test-classes [propertyfile] Creating new property file: /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/test-classes/docker-image.properties [INFO] Executed tasks [INFO] [INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ cbas-docker-test --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- docker-maven-plugin:7.0.0-SNAPSHOT:docker-prune (prune) @ cbas-docker-test --- [INFO] Removing docker objects created before: 2021-05-26T14:03:15.532102-07:00[America/Los_Angeles] (now()-PT336H) [WARNING] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. [WARNING] Cannot obtain created timestamp for network: tgip5ripmjjew1dmwz0c03epk. Skipping [INFO] [INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ cbas-docker-test --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 37 source files to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/test-classes [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[286,16] closeQuietly(java.io.Closeable) in org.apache.commons.io.IOUtils has been deprecated [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[529,35] withBinds(java.util.List) in com.github.dockerjava.api.command.CreateContainerCmd has been deprecated [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[531,35] withNetworkMode(java.lang.String) in com.github.dockerjava.api.command.CreateContainerCmd has been deprecated [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[534,35] withPortBindings(java.util.List) in com.github.dockerjava.api.command.CreateContainerCmd has been deprecated [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[537,35] withPublishAllPorts(java.lang.Boolean) in com.github.dockerjava.api.command.CreateContainerCmd has been deprecated [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[1076,68] com.github.dockerjava.core.command.AttachContainerResultCallback in com.github.dockerjava.core.command has been deprecated [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[1076,68] com.github.dockerjava.core.command.AttachContainerResultCallback in com.github.dockerjava.core.command has been deprecated [WARNING] /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/src/test/java/com/couchbase/analytics/test/docker/DockerTestBase.java:[1076,68] com.github.dockerjava.core.command.AttachContainerResultCallback in com.github.dockerjava.core.command has been deprecated [INFO] [INFO] --- maven-dependency-plugin:3.1.2:analyze-only (default) @ cbas-docker-test --- [INFO] No dependency problems found [INFO] [INFO] --- maven-resources-plugin:3.0.2:copy-resources (copy-docker-resources) @ cbas-docker-test --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 0 resource [INFO] [INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ cbas-docker-test --- [JENKINS] Recording test results [INFO] [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ cbas-docker-test --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] Building jar: /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/cbas-docker-test-7.0.0-SNAPSHOT.jar [INFO] [INFO] --- maven-jar-plugin:3.0.2:test-jar (default) @ cbas-docker-test --- [INFO] Building jar: /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/cbas-docker-test-7.0.0-SNAPSHOT-tests.jar [INFO] [INFO] --- maven-site-plugin:3.7.1:attach-descriptor (attach-descriptor) @ cbas-docker-test --- [INFO] Skipping because packaging 'jar' is not pom. [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:integration-test (default) @ cbas-docker-test --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running com.couchbase.analytics.test.docker.DockerClusterBasicITD 2021-06-09 14:03:20,519 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:03:20.613-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:03:20.655-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:03:20.656-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:03:20.666-07:00 WARN TestMethodTracer [main] ### twoNodeClusterDataAndAnalytics START 2021-06-09T14:03:20.711-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:03:22.079-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-116d35cc-75b1-46a4-9207-5bef2f3dc836) 2021-06-09T14:03:22.159-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-116d35cc-75b1-46a4-9207-5bef2f3dc836, id = f85be2be556e56bc85e26dea5215e3c0d753512879f494264cbaaa09277c731f } 2021-06-09T14:03:22.160-07:00 INFO DockerClusterBasicITD [main] Creating node #1 ... 2021-06-09T14:03:22.164-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:03:22.273-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:03:22.278-07:00 INFO DockerTestBase [main+] >> 345e3491a907: Pulling fs layer 2021-06-09T14:03:22.278-07:00 INFO DockerTestBase [main+] >> 57671312ef6f: Pulling fs layer 2021-06-09T14:03:22.279-07:00 INFO DockerTestBase [main+] >> 5e9250ddb7d0: Pulling fs layer 2021-06-09T14:03:22.279-07:00 INFO DockerTestBase [main+] >> dff68c104608: Pulling fs layer 2021-06-09T14:03:22.279-07:00 INFO DockerTestBase [main+] >> 5083acfb7755: Pulling fs layer 2021-06-09T14:03:22.279-07:00 INFO DockerTestBase [main+] >> de76a76007a1: Pulling fs layer 2021-06-09T14:03:22.280-07:00 INFO DockerTestBase [main+] >> 68d1c582f1c7: Pulling fs layer 2021-06-09T14:03:22.280-07:00 INFO DockerTestBase [main+] >> e4d81a4aec1a: Pulling fs layer 2021-06-09T14:03:22.280-07:00 INFO DockerTestBase [main+] >> 10fa2d157dbc: Pulling fs layer 2021-06-09T14:03:22.280-07:00 INFO DockerTestBase [main+] >> 6fa1a4e86a8c: Pulling fs layer 2021-06-09T14:03:22.281-07:00 INFO DockerTestBase [main+] >> c0b16e174e8a: Pulling fs layer 2021-06-09T14:03:22.281-07:00 INFO DockerTestBase [main+] >> 17e30cc5bf68: Pulling fs layer 2021-06-09T14:03:22.281-07:00 INFO DockerTestBase [main+] >> 8f2c9c1921b3: Pulling fs layer 2021-06-09T14:03:22.281-07:00 INFO DockerTestBase [main+] >> b603276b68d2: Pulling fs layer 2021-06-09T14:03:22.282-07:00 INFO DockerTestBase [main+] >> dff68c104608: Waiting 2021-06-09T14:03:22.282-07:00 INFO DockerTestBase [main+] >> 5083acfb7755: Waiting 2021-06-09T14:03:22.283-07:00 INFO DockerTestBase [main+] >> de76a76007a1: Waiting 2021-06-09T14:03:22.284-07:00 INFO DockerTestBase [main+] >> 68d1c582f1c7: Waiting 2021-06-09T14:03:22.285-07:00 INFO DockerTestBase [main+] >> e4d81a4aec1a: Waiting 2021-06-09T14:03:22.285-07:00 INFO DockerTestBase [main+] >> c0b16e174e8a: Waiting 2021-06-09T14:03:22.286-07:00 INFO DockerTestBase [main+] >> 10fa2d157dbc: Waiting 2021-06-09T14:03:22.286-07:00 INFO DockerTestBase [main+] >> 6fa1a4e86a8c: Waiting 2021-06-09T14:03:22.286-07:00 INFO DockerTestBase [main+] >> 17e30cc5bf68: Waiting 2021-06-09T14:03:22.287-07:00 INFO DockerTestBase [main+] >> 8f2c9c1921b3: Waiting 2021-06-09T14:03:22.287-07:00 INFO DockerTestBase [main+] >> b603276b68d2: Waiting 2021-06-09T14:03:22.337-07:00 INFO DockerTestBase [main+] >> 5e9250ddb7d0: Verifying Checksum 2021-06-09T14:03:22.337-07:00 INFO DockerTestBase [main+] >> 5e9250ddb7d0: Download complete 2021-06-09T14:03:22.337-07:00 INFO DockerTestBase [main+] >> 57671312ef6f: Verifying Checksum 2021-06-09T14:03:22.338-07:00 INFO DockerTestBase [main+] >> 57671312ef6f: Download complete 2021-06-09T14:03:22.374-07:00 INFO DockerTestBase [main+] >> 5083acfb7755: Download complete 2021-06-09T14:03:22.455-07:00 INFO DockerTestBase [main+] >> de76a76007a1: Verifying Checksum 2021-06-09T14:03:22.455-07:00 INFO DockerTestBase [main+] >> de76a76007a1: Download complete 2021-06-09T14:03:22.767-07:00 INFO DockerTestBase [main+] >> dff68c104608: Verifying Checksum 2021-06-09T14:03:22.767-07:00 INFO DockerTestBase [main+] >> dff68c104608: Download complete 2021-06-09T14:03:22.797-07:00 INFO DockerTestBase [main+] >> e4d81a4aec1a: Verifying Checksum 2021-06-09T14:03:22.797-07:00 INFO DockerTestBase [main+] >> e4d81a4aec1a: Download complete 2021-06-09T14:03:22.825-07:00 INFO DockerTestBase [main+] >> 10fa2d157dbc: Verifying Checksum 2021-06-09T14:03:22.825-07:00 INFO DockerTestBase [main+] >> 10fa2d157dbc: Download complete 2021-06-09T14:03:23.018-07:00 INFO DockerTestBase [main+] >> 6fa1a4e86a8c: Verifying Checksum 2021-06-09T14:03:23.018-07:00 INFO DockerTestBase [main+] >> 6fa1a4e86a8c: Download complete 2021-06-09T14:03:23.047-07:00 INFO DockerTestBase [main+] >> c0b16e174e8a: Verifying Checksum 2021-06-09T14:03:23.047-07:00 INFO DockerTestBase [main+] >> c0b16e174e8a: Download complete 2021-06-09T14:03:23.083-07:00 INFO DockerTestBase [main+] >> 17e30cc5bf68: Verifying Checksum 2021-06-09T14:03:23.083-07:00 INFO DockerTestBase [main+] >> 17e30cc5bf68: Download complete 2021-06-09T14:03:23.126-07:00 INFO DockerTestBase [main+] >> 8f2c9c1921b3: Verifying Checksum 2021-06-09T14:03:23.127-07:00 INFO DockerTestBase [main+] >> 8f2c9c1921b3: Download complete 2021-06-09T14:03:23.337-07:00 INFO DockerTestBase [main+] >> b603276b68d2: Verifying Checksum 2021-06-09T14:03:23.338-07:00 INFO DockerTestBase [main+] >> b603276b68d2: Download complete 2021-06-09T14:03:24.378-07:00 INFO DockerTestBase [main+] >> 345e3491a907: Verifying Checksum 2021-06-09T14:03:24.379-07:00 INFO DockerTestBase [main+] >> 345e3491a907: Download complete 2021-06-09T14:03:25.965-07:00 INFO DockerTestBase [main+] >> 345e3491a907: Pull complete 2021-06-09T14:03:26.114-07:00 INFO DockerTestBase [main+] >> 57671312ef6f: Pull complete 2021-06-09T14:03:26.257-07:00 INFO DockerTestBase [main+] >> 5e9250ddb7d0: Pull complete 2021-06-09T14:03:27.107-07:00 INFO DockerTestBase [main+] >> dff68c104608: Pull complete 2021-06-09T14:03:27.248-07:00 INFO DockerTestBase [main+] >> 5083acfb7755: Pull complete 2021-06-09T14:03:27.364-07:00 INFO DockerTestBase [main+] >> de76a76007a1: Pull complete 2021-06-09T14:03:35.189-07:00 INFO DockerTestBase [main+] >> 68d1c582f1c7: Verifying Checksum 2021-06-09T14:03:35.190-07:00 INFO DockerTestBase [main+] >> 68d1c582f1c7: Download complete 2021-06-09T14:03:52.228-07:00 INFO DockerTestBase [main+] >> 68d1c582f1c7: Pull complete 2021-06-09T14:03:53.761-07:00 INFO DockerTestBase [main+] >> e4d81a4aec1a: Pull complete 2021-06-09T14:03:55.134-07:00 INFO DockerTestBase [main+] >> 10fa2d157dbc: Pull complete 2021-06-09T14:03:55.820-07:00 INFO DockerTestBase [main+] >> 6fa1a4e86a8c: Pull complete 2021-06-09T14:03:56.742-07:00 INFO DockerTestBase [main+] >> c0b16e174e8a: Pull complete 2021-06-09T14:03:57.948-07:00 INFO DockerTestBase [main+] >> 17e30cc5bf68: Pull complete 2021-06-09T14:03:59.553-07:00 INFO DockerTestBase [main+] >> 8f2c9c1921b3: Pull complete 2021-06-09T14:03:59.663-07:00 INFO DockerTestBase [main+] >> b603276b68d2: Pull complete 2021-06-09T14:03:59.670-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:03:59.674-07:00 INFO DockerTestBase [main+] >> Status: Downloaded newer image for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:03:59.676-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:03:59.679-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.DockerClusterBasicITD/twoNodeClusterDataAndAnalytics/0001_n1_thisbuild, network = { name = cbas-docker-test-0001-116d35cc-75b1-46a4-9207-5bef2f3dc836, id = f85be2be556e56bc85e26dea5215e3c0d753512879f494264cbaaa09277c731f }) 2021-06-09T14:03:59.700-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:38714/tcp 2021-06-09T14:03:59.701-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:33659/tcp 2021-06-09T14:03:59.848-07:00 INFO DockerTestBase [main] Starting docker container: 4d2cbd8ae67b6 2021-06-09T14:04:00.658-07:00 INFO DockerTestBase [main] Started docker container: 4d2cbd8ae67b6 2021-06-09T14:04:00.876-07:00 INFO DockerTestBase [main] Container information for 4d2cbd8ae67b6: host: n1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:33659->192.168.176.2:8095, 172.17.0.1:38714->192.168.176.2:8091 2021-06-09T14:04:11.000-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:04:11.002-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:04:12.281-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:04:12.324-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T14:04:12.325-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:04:12.835-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:04:12.886-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:04:27.564-07:00 INFO DockerClusterBasicITD [main] Creating node #2 ... 2021-06-09T14:04:27.565-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.DockerClusterBasicITD/twoNodeClusterDataAndAnalytics/0002_n2_thisbuild, network = { name = cbas-docker-test-0001-116d35cc-75b1-46a4-9207-5bef2f3dc836, id = f85be2be556e56bc85e26dea5215e3c0d753512879f494264cbaaa09277c731f }) 2021-06-09T14:04:27.566-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:34444/tcp 2021-06-09T14:04:27.566-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:38660/tcp 2021-06-09T14:04:27.612-07:00 INFO DockerTestBase [main] Starting docker container: 3ea8880dcb88e 2021-06-09T14:04:28.415-07:00 INFO DockerTestBase [main] Started docker container: 3ea8880dcb88e 2021-06-09T14:04:28.430-07:00 INFO DockerTestBase [main] Container information for 3ea8880dcb88e: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:38660->192.168.176.3:8095, 172.17.0.1:34444->192.168.176.3:8091 2021-06-09T14:04:38.482-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:04:38.483-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:04:39.292-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:04:39.331-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:04:39.332-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data,analytics 2021-06-09T14:04:43.457-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:04:43.483-07:00 INFO DockerClusterBasicITD [main] Rebalancing ... 2021-06-09T14:04:43.484-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:04:43.484-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:04:54.991-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:04:54.992-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:04:55.025-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:04:55.057-07:00 INFO DockerTestBase [main] Copying files from docker container 3ea8880dcb88e: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.DockerClusterBasicITD/twoNodeClusterDataAndAnalytics/0002_n2_thisbuild 2021-06-09T14:04:55.058-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 4d2cbd8ae67b6: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.DockerClusterBasicITD/twoNodeClusterDataAndAnalytics/0001_n1_thisbuild 2021-06-09T14:04:55.345-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 4d2cbd8ae67b6 2021-06-09T14:04:55.379-07:00 INFO DockerTestBase [main] Stopping docker container: 3ea8880dcb88e 2021-06-09T14:04:55.910-07:00 INFO DockerTestBase [main] Stopped docker container: 3ea8880dcb88e 2021-06-09T14:04:55.951-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 4d2cbd8ae67b6 2021-06-09T14:04:55.954-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 4d2cbd8ae67b6 2021-06-09T14:04:55.965-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 4d2cbd8ae67b6 2021-06-09T14:04:55.954-07:00 INFO DockerTestBase [main] Removing docker container: 3ea8880dcb88e 2021-06-09T14:04:55.976-07:00 INFO DockerTestBase [main] Removed docker container: 3ea8880dcb88e 2021-06-09T14:04:55.977-07:00 INFO DockerTestBase [main] Removing docker network: f85be2be556e56bc85e26dea5215e3c0d753512879f494264cbaaa09277c731f 2021-06-09T14:04:56.059-07:00 INFO DockerTestBase [main] Removed docker network: f85be2be556e56bc85e26dea5215e3c0d753512879f494264cbaaa09277c731f 2021-06-09T14:04:56.065-07:00 WARN TestMethodTracer [main] ### twoNodeClusterDataAndAnalytics SUCCEEDED 2021-06-09T14:04:56.066-07:00 WARN TestMethodTracer [main] ### verifyGMTOutputInLogs START 2021-06-09T14:04:56.068-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:04:56.120-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0002-a050cc31-98f6-45a7-8628-4874e1d10455) 2021-06-09T14:04:56.171-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0002-a050cc31-98f6-45a7-8628-4874e1d10455, id = c54b1b9809adee8aad3ede1b231a641a43ef326c3e049833bbb6afd43cb85efe } 2021-06-09T14:04:56.171-07:00 INFO DockerClusterBasicITD [main] Creating node ... 2021-06-09T14:04:56.172-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.DockerClusterBasicITD/verifyGMTOutputInLogs/0003_n1_thisbuild, network = { name = cbas-docker-test-0002-a050cc31-98f6-45a7-8628-4874e1d10455, id = c54b1b9809adee8aad3ede1b231a641a43ef326c3e049833bbb6afd43cb85efe }) 2021-06-09T14:04:56.175-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:35868/tcp 2021-06-09T14:04:56.176-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:39684/tcp 2021-06-09T14:04:56.234-07:00 INFO DockerTestBase [main] Starting docker container: 3e19295ede001 2021-06-09T14:04:56.990-07:00 INFO DockerTestBase [main] Started docker container: 3e19295ede001 2021-06-09T14:04:57.034-07:00 INFO DockerTestBase [main] Container information for 3e19295ede001: host: n1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:39684->192.168.176.2:8095, 172.17.0.1:35868->192.168.176.2:8091 2021-06-09T14:05:07.078-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:05:07.079-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:05:07.933-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:05:07.980-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T14:05:07.980-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:05:08.501-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:05:08.535-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:05:22.170-07:00 INFO DockerTestBase [main] Copying files from docker container 3e19295ede001: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.DockerClusterBasicITD/verifyGMTOutputInLogs/0003_n1_thisbuild 2021-06-09T14:05:22.433-07:00 INFO DockerTestBase [main] Stopping docker container: 3e19295ede001 2021-06-09T14:05:22.873-07:00 INFO DockerTestBase [main] Stopped docker container: 3e19295ede001 2021-06-09T14:05:22.925-07:00 INFO DockerTestBase [main] Removing docker container: 3e19295ede001 2021-06-09T14:05:22.937-07:00 INFO DockerTestBase [main] Removed docker container: 3e19295ede001 2021-06-09T14:05:22.938-07:00 INFO DockerTestBase [main] Removing docker network: c54b1b9809adee8aad3ede1b231a641a43ef326c3e049833bbb6afd43cb85efe 2021-06-09T14:05:23.023-07:00 INFO DockerTestBase [main] Removed docker network: c54b1b9809adee8aad3ede1b231a641a43ef326c3e049833bbb6afd43cb85efe 2021-06-09T14:05:23.025-07:00 WARN TestMethodTracer [main] ### verifyGMTOutputInLogs SUCCEEDED [WARNING] Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 123.86 s - in com.couchbase.analytics.test.docker.DockerClusterBasicITD [INFO] Running com.couchbase.analytics.test.docker.azure.AzureBestPracticesITD 2021-06-09 14:05:25,996 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:05:26.145-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:05:26.201-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:05:26.201-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:05:26.210-07:00 WARN TestMethodTracer [main] ### testLoopbackPublicAddress START 2021-06-09T14:05:26.256-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:05:27.591-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-62c57f3a-7317-4f3e-b240-213ba3ab0b1c) 2021-06-09T14:05:27.707-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-62c57f3a-7317-4f3e-b240-213ba3ab0b1c, id = 298c96bdd0bd802a9f144754273f12a094b65b240a2de61d8eeb2a000c51532c } 2021-06-09T14:05:27.711-07:00 INFO AzureBestPracticesITD [pool-0] Creating n1 node ... 2021-06-09T14:05:27.720-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:05:27.721-07:00 INFO AzureBestPracticesITD [pool-1] Creating n2 node ... 2021-06-09T14:05:27.722-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:05:27.744-07:00 INFO AzureBestPracticesITD [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/hosts.n1 to /etc/hosts 2021-06-09T14:05:27.744-07:00 INFO AzureBestPracticesITD [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/hosts.n2 to /etc/hosts 2021-06-09T14:05:27.747-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:05:27.903-07:00 INFO DockerTestBase [pool-0+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:05:27.924-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:05:27.925-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:05:27.945-07:00 INFO DockerTestBase [pool-0+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:05:27.950-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.azure.AzureBestPracticesITD/testLoopbackPublicAddress/0002_n2_thisbuild, network = { name = cbas-docker-test-0001-62c57f3a-7317-4f3e-b240-213ba3ab0b1c, id = 298c96bdd0bd802a9f144754273f12a094b65b240a2de61d8eeb2a000c51532c }) 2021-06-09T14:05:27.950-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.azure.AzureBestPracticesITD/testLoopbackPublicAddress/0001_n1_thisbuild, network = { name = cbas-docker-test-0001-62c57f3a-7317-4f3e-b240-213ba3ab0b1c, id = 298c96bdd0bd802a9f144754273f12a094b65b240a2de61d8eeb2a000c51532c }) 2021-06-09T14:05:27.994-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8091/tcp -> 172.17.0.1:40799/tcp 2021-06-09T14:05:27.994-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8091/tcp -> 172.17.0.1:41540/tcp 2021-06-09T14:05:27.995-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8095/tcp -> 172.17.0.1:44633/tcp 2021-06-09T14:05:27.995-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8095/tcp -> 172.17.0.1:43378/tcp 2021-06-09T14:05:28.332-07:00 INFO DockerTestBase [pool-1] Starting docker container: e8c0591e999e0 2021-06-09T14:05:28.368-07:00 INFO DockerTestBase [pool-0] Starting docker container: 21ef3cc375e67 2021-06-09T14:05:29.451-07:00 INFO DockerTestBase [pool-1] Started docker container: e8c0591e999e0 2021-06-09T14:05:29.514-07:00 INFO DockerTestBase [pool-0] Started docker container: 21ef3cc375e67 2021-06-09T14:05:29.706-07:00 INFO DockerTestBase [pool-1] Container information for e8c0591e999e0: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:44633->192.168.176.2:8095, 172.17.0.1:40799->192.168.176.2:8091 2021-06-09T14:05:29.722-07:00 INFO DockerTestBase [pool-0] Container information for 21ef3cc375e67: host: n1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:43378->192.168.176.3:8095, 172.17.0.1:41540->192.168.176.3:8091 2021-06-09T14:05:40.937-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1 (192.168.176.3) 2021-06-09T14:05:40.940-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:05:40.960-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node n2 (192.168.176.2) 2021-06-09T14:05:40.961-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:05:42.069-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:05:42.071-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:05:42.124-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host sh -c cat /etc/hosts 2021-06-09T14:05:42.272-07:00 INFO AzureBestPracticesITD [main] hosts file: 127.0.0.1 locahost n1 n1.couchbase.host 2021-06-09T14:05:42.281-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:05:42.282-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:05:42.788-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:05:42.818-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.2) with services: [ANALYTICS] 2021-06-09T14:05:42.819-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:05:47.276-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:05:47.312-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:05:47.313-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase 2021-06-09T14:06:08.006-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:06:08.007-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:06:08.050-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:06:08.066-07:00 INFO DockerTestBase [pool-1] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:06:08.067-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:06:08.074-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:06:08.346-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:41540] 2021-06-09T14:06:08.350-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:41540 2021-06-09T14:06:09.414-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:40799] 2021-06-09T14:06:14.777-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:06:14.780-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:06:15.715-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:06:15.720-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:06:16.722-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:06:17.195-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2936 } 2021-06-09T14:06:17.195-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2936 } 2021-06-09T14:06:18.196-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:06:18.375-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:06:18.375-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:06:18.378-07:00 INFO DockerTestBase [main] Copying files from docker container 21ef3cc375e67: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.azure.AzureBestPracticesITD/testLoopbackPublicAddress/0001_n1_thisbuild 2021-06-09T14:06:18.379-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container e8c0591e999e0: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.azure.AzureBestPracticesITD/testLoopbackPublicAddress/0002_n2_thisbuild 2021-06-09T14:06:18.797-07:00 INFO DockerTestBase [main] Stopping docker container: 21ef3cc375e67 2021-06-09T14:06:18.878-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: e8c0591e999e0 2021-06-09T14:06:19.512-07:00 INFO DockerTestBase [main] Stopped docker container: 21ef3cc375e67 2021-06-09T14:06:19.554-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: e8c0591e999e0 2021-06-09T14:06:19.556-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: e8c0591e999e0 2021-06-09T14:06:19.556-07:00 INFO DockerTestBase [main] Removing docker container: 21ef3cc375e67 2021-06-09T14:06:19.567-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: e8c0591e999e0 2021-06-09T14:06:19.572-07:00 INFO DockerTestBase [main] Removed docker container: 21ef3cc375e67 2021-06-09T14:06:19.574-07:00 INFO DockerTestBase [main] Removing docker network: 298c96bdd0bd802a9f144754273f12a094b65b240a2de61d8eeb2a000c51532c 2021-06-09T14:06:19.640-07:00 INFO DockerTestBase [main] Removed docker network: 298c96bdd0bd802a9f144754273f12a094b65b240a2de61d8eeb2a000c51532c 2021-06-09T14:06:19.647-07:00 WARN TestMethodTracer [main] ### testLoopbackPublicAddress SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.386 s - in com.couchbase.analytics.test.docker.azure.AzureBestPracticesITD [INFO] Running com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetDPDisabledITD 2021-06-09 14:06:22,254 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:06:22.347-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:06:22.652-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:06:22.653-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:06:22.704-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:06:23.855-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-9443a46f-637a-41ce-98b1-fbef91fe665e) 2021-06-09T14:06:23.945-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-9443a46f-637a-41ce-98b1-fbef91fe665e, id = 92c3f214089d3cf5f7bb08f2781a54c69e2402165197ab6c759cfa1ce82be51f } 2021-06-09T14:06:23.948-07:00 INFO DockerTestBase [main] Executing the command: docker pull mcr.microsoft.com/azure-storage/azurite:3.9.0 2021-06-09T14:06:24.890-07:00 INFO DockerTestBase [main+] >> 3.9.0: Pulling from azure-storage/azurite 2021-06-09T14:06:24.891-07:00 INFO DockerTestBase [main+] >> cbdbe7a5bc2a: Pulling fs layer 2021-06-09T14:06:24.891-07:00 INFO DockerTestBase [main+] >> 9287919c3a0f: Pulling fs layer 2021-06-09T14:06:24.891-07:00 INFO DockerTestBase [main+] >> 43a47bbd54c9: Pulling fs layer 2021-06-09T14:06:24.892-07:00 INFO DockerTestBase [main+] >> 3c1bcea295c4: Pulling fs layer 2021-06-09T14:06:24.892-07:00 INFO DockerTestBase [main+] >> 5047118b6920: Pulling fs layer 2021-06-09T14:06:24.892-07:00 INFO DockerTestBase [main+] >> 77c92003e19f: Pulling fs layer 2021-06-09T14:06:24.892-07:00 INFO DockerTestBase [main+] >> 13ec4a55a3c3: Pulling fs layer 2021-06-09T14:06:24.892-07:00 INFO DockerTestBase [main+] >> 21145ef27392: Pulling fs layer 2021-06-09T14:06:24.893-07:00 INFO DockerTestBase [main+] >> db43acfda40e: Pulling fs layer 2021-06-09T14:06:24.893-07:00 INFO DockerTestBase [main+] >> 77c92003e19f: Waiting 2021-06-09T14:06:24.893-07:00 INFO DockerTestBase [main+] >> 13ec4a55a3c3: Waiting 2021-06-09T14:06:24.893-07:00 INFO DockerTestBase [main+] >> 21145ef27392: Waiting 2021-06-09T14:06:24.893-07:00 INFO DockerTestBase [main+] >> 3c1bcea295c4: Waiting 2021-06-09T14:06:24.894-07:00 INFO DockerTestBase [main+] >> db43acfda40e: Waiting 2021-06-09T14:06:24.894-07:00 INFO DockerTestBase [main+] >> 5047118b6920: Waiting 2021-06-09T14:06:25.430-07:00 INFO DockerTestBase [main+] >> 43a47bbd54c9: Verifying Checksum 2021-06-09T14:06:25.431-07:00 INFO DockerTestBase [main+] >> 43a47bbd54c9: Download complete 2021-06-09T14:06:25.605-07:00 INFO DockerTestBase [main+] >> 9287919c3a0f: Verifying Checksum 2021-06-09T14:06:25.606-07:00 INFO DockerTestBase [main+] >> 9287919c3a0f: Download complete 2021-06-09T14:06:25.680-07:00 INFO DockerTestBase [main+] >> 3c1bcea295c4: Verifying Checksum 2021-06-09T14:06:25.681-07:00 INFO DockerTestBase [main+] >> 3c1bcea295c4: Download complete 2021-06-09T14:06:25.957-07:00 INFO DockerTestBase [main+] >> cbdbe7a5bc2a: Download complete 2021-06-09T14:06:26.148-07:00 INFO DockerTestBase [main+] >> 5047118b6920: Verifying Checksum 2021-06-09T14:06:26.148-07:00 INFO DockerTestBase [main+] >> 5047118b6920: Download complete 2021-06-09T14:06:26.190-07:00 INFO DockerTestBase [main+] >> cbdbe7a5bc2a: Pull complete 2021-06-09T14:06:26.412-07:00 INFO DockerTestBase [main+] >> 77c92003e19f: Verifying Checksum 2021-06-09T14:06:26.412-07:00 INFO DockerTestBase [main+] >> 77c92003e19f: Download complete 2021-06-09T14:06:26.499-07:00 INFO DockerTestBase [main+] >> 21145ef27392: Download complete 2021-06-09T14:06:27.011-07:00 INFO DockerTestBase [main+] >> db43acfda40e: Verifying Checksum 2021-06-09T14:06:27.011-07:00 INFO DockerTestBase [main+] >> db43acfda40e: Download complete 2021-06-09T14:06:28.064-07:00 INFO DockerTestBase [main+] >> 9287919c3a0f: Pull complete 2021-06-09T14:06:28.258-07:00 INFO DockerTestBase [main+] >> 43a47bbd54c9: Pull complete 2021-06-09T14:06:28.364-07:00 INFO DockerTestBase [main+] >> 3c1bcea295c4: Pull complete 2021-06-09T14:06:28.476-07:00 INFO DockerTestBase [main+] >> 5047118b6920: Pull complete 2021-06-09T14:06:28.629-07:00 INFO DockerTestBase [main+] >> 77c92003e19f: Pull complete 2021-06-09T14:06:29.922-07:00 INFO DockerTestBase [main+] >> 13ec4a55a3c3: Verifying Checksum 2021-06-09T14:06:29.923-07:00 INFO DockerTestBase [main+] >> 13ec4a55a3c3: Download complete 2021-06-09T14:06:31.397-07:00 INFO DockerTestBase [main+] >> 13ec4a55a3c3: Pull complete 2021-06-09T14:06:31.653-07:00 INFO DockerTestBase [main+] >> 21145ef27392: Pull complete 2021-06-09T14:06:31.785-07:00 INFO DockerTestBase [main+] >> db43acfda40e: Pull complete 2021-06-09T14:06:31.791-07:00 INFO DockerTestBase [main+] >> Digest: sha256:6b6c9c24e97a6c000a81503a41bbf366f04c23aa3b473bfc27267145c92eeceb 2021-06-09T14:06:31.794-07:00 INFO DockerTestBase [main+] >> Status: Downloaded newer image for mcr.microsoft.com/azure-storage/azurite:3.9.0 2021-06-09T14:06:31.796-07:00 INFO DockerTestBase [main+] >> mcr.microsoft.com/azure-storage/azurite:3.9.0 2021-06-09T14:06:31.800-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0001_azurite_mcr.microsoft.com_azure-storage_azurite_3.9.0, network = { name = cbas-docker-test-0001-9443a46f-637a-41ce-98b1-fbef91fe665e, id = 92c3f214089d3cf5f7bb08f2781a54c69e2402165197ab6c759cfa1ce82be51f }) 2021-06-09T14:06:31.823-07:00 INFO DockerTestBase [main] binding exposed port azurite:10000/tcp -> 172.17.0.1:35009/tcp 2021-06-09T14:06:31.824-07:00 INFO DockerTestBase [main] binding exposed port azurite:8091/tcp -> 172.17.0.1:38840/tcp 2021-06-09T14:06:31.824-07:00 INFO DockerTestBase [main] binding exposed port azurite:8095/tcp -> 172.17.0.1:41191/tcp 2021-06-09T14:06:31.947-07:00 INFO DockerTestBase [main] Starting docker container: 8bbe16f189265 2021-06-09T14:06:32.515-07:00 INFO DockerTestBase [main] Started docker container: 8bbe16f189265 2021-06-09T14:06:32.664-07:00 INFO DockerTestBase [main] Container information for 8bbe16f189265: host: azurite.couchbase.host, image: 3.9.0, ip: 192.168.176.2, 172.17.0.1:35009->192.168.176.2:10000, 172.17.0.1:38840->192.168.176.2:8091, 172.17.0.1:41191->192.168.176.2:8095 2021-06-09T14:06:32.665-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:06:32.773-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:06:32.775-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:06:32.776-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:06:32.778-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:06:32.781-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0002_cbas_thisbuild, network = { name = cbas-docker-test-0001-9443a46f-637a-41ce-98b1-fbef91fe665e, id = 92c3f214089d3cf5f7bb08f2781a54c69e2402165197ab6c759cfa1ce82be51f }) 2021-06-09T14:06:32.782-07:00 INFO DockerTestBase [main] binding exposed port cbas:8091/tcp -> 172.17.0.1:36196/tcp 2021-06-09T14:06:32.783-07:00 INFO DockerTestBase [main] binding exposed port cbas:8095/tcp -> 172.17.0.1:41752/tcp 2021-06-09T14:06:32.840-07:00 INFO DockerTestBase [main] Starting docker container: 95534c1eb63b6 2021-06-09T14:06:33.640-07:00 INFO DockerTestBase [main] Started docker container: 95534c1eb63b6 2021-06-09T14:06:33.661-07:00 INFO DockerTestBase [main] Container information for 95534c1eb63b6: host: cbas.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:36196->192.168.176.3:8091, 172.17.0.1:41752->192.168.176.3:8095 2021-06-09T14:06:44.107-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36196] 2021-06-09T14:06:44.118-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node cbas (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:06:44.120-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:06:44.705-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:06:44.746-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:06:59.220-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating Azurite Blob Service client 2021-06-09T14:06:59.825-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting container playground 2021-06-09T14:07:00.697-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Container playground deleted successfully 2021-06-09T14:07:00.698-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting container fixed-data 2021-06-09T14:07:00.718-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Container fixed-data deleted successfully 2021-06-09T14:07:00.718-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting container include-exclude 2021-06-09T14:07:00.761-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Container include-exclude deleted successfully 2021-06-09T14:07:00.850-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Azurite Blob Service client created successfully 2021-06-09T14:07:00.866-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetDPDisabledITD 0: external-dataset/azure_blob_storage/: dp_disabled] START 2021-06-09T14:07:00.949-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage//dp_disabled ... 2021-06-09T14:07:01.081-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/dp_disabled/result.000.txt 2021-06-09T14:07:01.084-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:07:01.482-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/azure_blob_storage/dp_disabled/test.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:07:01.483-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage//dp_disabled PASSED 2021-06-09T14:07:01.485-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/: dp_disabled... 2021-06-09T14:07:02.157-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:07:02.768-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/: dp_disabled 2021-06-09T14:07:02.769-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetDPDisabledITD 0: external-dataset/azure_blob_storage/: dp_disabled] SUCCEEDED 2021-06-09T14:07:02.774-07:00 INFO DockerTestBase [main] Copying files from docker container 95534c1eb63b6: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0002_cbas_thisbuild 2021-06-09T14:07:02.774-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 8bbe16f189265: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0001_azurite_mcr.microsoft.com_azure-storage_azurite_3.9.0 2021-06-09T14:07:02.867-07:00 WARN DockerTestBase [ForkJoinPool.commonPool-worker-3] Error copying files from docker container: 8bbe16f189265 com.github.dockerjava.api.exception.NotFoundException: Status 404: Could not find the file /opt/couchbase/var/lib/couchbase/logs/ in container 8bbe16f18926513399ab1356fd2e949559d5d7269e5213dcbee3e8ea4226b4fb at com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:62) ~[docker-java-transport-jersey-3.2.8.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608) ~[jersey-client-2.30.1.jar:?] at com.github.dockerjava.jaxrs.JerseyDockerHttpClient.execute(JerseyDockerHttpClient.java:297) ~[docker-java-transport-jersey-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.CopyArchiveFromContainerCmdExec.execute(CopyArchiveFromContainerCmdExec.java:30) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.CopyArchiveFromContainerCmdExec.execute(CopyArchiveFromContainerCmdExec.java:13) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.CopyArchiveFromContainerCmdImpl.exec(CopyArchiveFromContainerCmdImpl.java:70) ~[docker-java-core-3.2.8.jar:?] at com.couchbase.analytics.test.docker.DockerTestBase.copyFilesFromContainer(DockerTestBase.java:1008) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.stopContainer(DockerTestBase.java:875) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.stopContainer(DockerTestBase.java:856) ~[test-classes/:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) [?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) [?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?] at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) [?:?] at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) [?:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?] at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?] at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?] 2021-06-09T14:07:02.880-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 8bbe16f189265 2021-06-09T14:07:03.146-07:00 INFO DockerTestBase [main] Stopping docker container: 95534c1eb63b6 2021-06-09T14:07:04.110-07:00 INFO DockerTestBase [main] Stopped docker container: 95534c1eb63b6 2021-06-09T14:07:06.077-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 8bbe16f189265 2021-06-09T14:07:06.079-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 8bbe16f189265 2021-06-09T14:07:06.079-07:00 INFO DockerTestBase [main] Removing docker container: 95534c1eb63b6 2021-06-09T14:07:06.090-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 8bbe16f189265 2021-06-09T14:07:06.095-07:00 INFO DockerTestBase [main] Removed docker container: 95534c1eb63b6 2021-06-09T14:07:06.096-07:00 INFO DockerTestBase [main] Removing docker network: 92c3f214089d3cf5f7bb08f2781a54c69e2402165197ab6c759cfa1ce82be51f 2021-06-09T14:07:06.161-07:00 INFO DockerTestBase [main] Removed docker network: 92c3f214089d3cf5f7bb08f2781a54c69e2402165197ab6c759cfa1ce82be51f [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 45.499 s - in com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetDPDisabledITD [INFO] Running com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD 2021-06-09 14:07:08,522 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:07:08.604-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:07:08.972-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:07:08.972-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:07:09.025-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:07:10.178-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-c4edd74c-b246-445a-8f8e-b91c0143bf58) 2021-06-09T14:07:10.250-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-c4edd74c-b246-445a-8f8e-b91c0143bf58, id = d49f88ec6d1f62995f0c08be9babdd8587f00b53224dee0219795721f6b6ac77 } 2021-06-09T14:07:10.253-07:00 INFO DockerTestBase [main] Executing the command: docker pull mcr.microsoft.com/azure-storage/azurite:3.9.0 2021-06-09T14:07:10.656-07:00 INFO DockerTestBase [main+] >> 3.9.0: Pulling from azure-storage/azurite 2021-06-09T14:07:10.657-07:00 INFO DockerTestBase [main+] >> Digest: sha256:6b6c9c24e97a6c000a81503a41bbf366f04c23aa3b473bfc27267145c92eeceb 2021-06-09T14:07:10.658-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for mcr.microsoft.com/azure-storage/azurite:3.9.0 2021-06-09T14:07:10.659-07:00 INFO DockerTestBase [main+] >> mcr.microsoft.com/azure-storage/azurite:3.9.0 2021-06-09T14:07:10.662-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0001_azurite_mcr.microsoft.com_azure-storage_azurite_3.9.0, network = { name = cbas-docker-test-0001-c4edd74c-b246-445a-8f8e-b91c0143bf58, id = d49f88ec6d1f62995f0c08be9babdd8587f00b53224dee0219795721f6b6ac77 }) 2021-06-09T14:07:10.684-07:00 INFO DockerTestBase [main] binding exposed port azurite:10000/tcp -> 172.17.0.1:36005/tcp 2021-06-09T14:07:10.686-07:00 INFO DockerTestBase [main] binding exposed port azurite:8091/tcp -> 172.17.0.1:33153/tcp 2021-06-09T14:07:10.686-07:00 INFO DockerTestBase [main] binding exposed port azurite:8095/tcp -> 172.17.0.1:38485/tcp 2021-06-09T14:07:10.797-07:00 INFO DockerTestBase [main] Starting docker container: 5aa089d97ea5a 2021-06-09T14:07:11.386-07:00 INFO DockerTestBase [main] Started docker container: 5aa089d97ea5a 2021-06-09T14:07:11.592-07:00 INFO DockerTestBase [main] Container information for 5aa089d97ea5a: host: azurite.couchbase.host, image: 3.9.0, ip: 192.168.176.2, 172.17.0.1:33153->192.168.176.2:8091, 172.17.0.1:38485->192.168.176.2:8095, 172.17.0.1:36005->192.168.176.2:10000 2021-06-09T14:07:11.593-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:07:11.715-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:07:11.716-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:07:11.716-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:07:11.718-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:07:11.724-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0002_cbas_thisbuild, network = { name = cbas-docker-test-0001-c4edd74c-b246-445a-8f8e-b91c0143bf58, id = d49f88ec6d1f62995f0c08be9babdd8587f00b53224dee0219795721f6b6ac77 }) 2021-06-09T14:07:11.725-07:00 INFO DockerTestBase [main] binding exposed port cbas:8091/tcp -> 172.17.0.1:45023/tcp 2021-06-09T14:07:11.725-07:00 INFO DockerTestBase [main] binding exposed port cbas:8095/tcp -> 172.17.0.1:36193/tcp 2021-06-09T14:07:11.789-07:00 INFO DockerTestBase [main] Starting docker container: c5d85d4db0752 2021-06-09T14:07:12.599-07:00 INFO DockerTestBase [main] Started docker container: c5d85d4db0752 2021-06-09T14:07:12.607-07:00 INFO DockerTestBase [main] Container information for c5d85d4db0752: host: cbas.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:36193->192.168.176.3:8095, 172.17.0.1:45023->192.168.176.3:8091 2021-06-09T14:07:22.760-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:45023] 2021-06-09T14:07:22.770-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node cbas (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:07:22.772-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:07:23.354-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:07:23.408-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:07:37.012-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] +http://172.17.0.1:45023/settings/developerPreview 2021-06-09T14:07:37.031-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Waiting for 'developer preview' condition for up to 30 seconds... 2021-06-09T14:07:37.053-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Developer Preview enabled successfully. 2021-06-09T14:07:37.054-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Waiting for 10 seconds for Analytics Driver restart to happen... 2021-06-09T14:07:47.054-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:07:49.214-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating Azurite Blob Service client 2021-06-09T14:07:49.710-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting container playground 2021-06-09T14:07:50.344-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Container playground deleted successfully 2021-06-09T14:07:50.344-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting container fixed-data 2021-06-09T14:07:50.368-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Container fixed-data deleted successfully 2021-06-09T14:07:50.369-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting container include-exclude 2021-06-09T14:07:50.382-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Container include-exclude deleted successfully 2021-06-09T14:07:50.464-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Azurite Blob Service client created successfully 2021-06-09T14:07:50.474-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] +http://172.17.0.1:36193/analytics/service 2021-06-09T14:07:50.798-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:50.799-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope Default --name connectionStringAccountNameAccountKeyLink --type azureblob --connection-string AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://172.17.0.1:36005/devstoreaccount1 2021-06-09T14:07:51.470-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Link created 2021-06-09T14:07:51.494-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:51.495-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope Default --name connectionStringAccountNameSasLink --type azureblob --connection-string AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://172.17.0.1:36005/devstoreaccount1 2021-06-09T14:07:51.857-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Link created 2021-06-09T14:07:51.884-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:51.884-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope linksDataverse --name accountNameAccountKey --type azureblob --account-name devstoreaccount1 --account-key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== 2021-06-09T14:07:52.250-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Link created 2021-06-09T14:07:52.276-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:52.276-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope linksDataverse --name accountNameSas --type azureblob --account-name devstoreaccount1 --shared-access-signature sv=2019-07-07&ss=b&srt=sco&se=2022-06-09T21%3A07%3A50Z&sp=rwdlacup&sig=WO9E%2F83qhfL9TSNr%2BkYxEaNncbE8TwBca7Zm7aBVNlI%3D 2021-06-09T14:07:52.625-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Link created 2021-06-09T14:07:52.656-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:52.657-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope Default --name noAuthLink --type azureblob 2021-06-09T14:07:53.002-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> ERROR: No authentication parameters provided 2021-06-09T14:07:53.029-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:53.029-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope Default --name connectionStringAccountNameAccountKeyAndAccountKeyLink --type azureblob --connection-string AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://172.17.0.1:36005/devstoreaccount1 --account-key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== 2021-06-09T14:07:53.376-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> ERROR: Only a single authentication method is allowed 2021-06-09T14:07:53.406-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:53.407-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope Default --name connectionStringAccountNameAccountKeyAndSasLink --type azureblob --connection-string AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://172.17.0.1:36005/devstoreaccount1 --account-key sv=2019-07-07&ss=b&srt=sco&se=2022-06-09T21%3A07%3A50Z&sp=rwdlacup&sig=WO9E%2F83qhfL9TSNr%2BkYxEaNncbE8TwBca7Zm7aBVNlI%3D 2021-06-09T14:07:53.757-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> ERROR: Only a single authentication method is allowed 2021-06-09T14:07:53.782-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Creating AzureBlob link... 2021-06-09T14:07:53.782-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --scope Default --name accountNameAccountKeyAndSasLink --type azureblob --account-name devstoreaccount1 --account-key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== --shared-access-signature sv=2019-07-07&ss=b&srt=sco&se=2022-06-09T21%3A07%3A50Z&sp=rwdlacup&sig=WO9E%2F83qhfL9TSNr%2BkYxEaNncbE8TwBca7Zm7aBVNlI%3D 2021-06-09T14:07:54.109-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> ERROR: Only a single authentication method is allowed 2021-06-09T14:07:54.133-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --list --type azureblob 2021-06-09T14:07:54.524-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> [ 2021-06-09T14:07:54.524-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> { 2021-06-09T14:07:54.524-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountKey": null, 2021-06-09T14:07:54.525-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountName": null, 2021-06-09T14:07:54.525-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "blobEndpoint": null, 2021-06-09T14:07:54.525-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "connectionString": "", 2021-06-09T14:07:54.525-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "endpointSuffix": null, 2021-06-09T14:07:54.526-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "name": "connectionStringAccountNameAccountKeyLink", 2021-06-09T14:07:54.526-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "scope": "Default", 2021-06-09T14:07:54.526-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "sharedAccessSignature": null, 2021-06-09T14:07:54.526-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "type": "azureblob" 2021-06-09T14:07:54.526-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> }, 2021-06-09T14:07:54.527-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> { 2021-06-09T14:07:54.527-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountKey": null, 2021-06-09T14:07:54.527-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountName": null, 2021-06-09T14:07:54.527-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "blobEndpoint": null, 2021-06-09T14:07:54.528-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "connectionString": "", 2021-06-09T14:07:54.528-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "endpointSuffix": null, 2021-06-09T14:07:54.528-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "name": "connectionStringAccountNameSasLink", 2021-06-09T14:07:54.528-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "scope": "Default", 2021-06-09T14:07:54.529-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "sharedAccessSignature": null, 2021-06-09T14:07:54.529-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "type": "azureblob" 2021-06-09T14:07:54.529-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> }, 2021-06-09T14:07:54.529-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> { 2021-06-09T14:07:54.529-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountKey": "", 2021-06-09T14:07:54.530-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountName": "devstoreaccount1", 2021-06-09T14:07:54.530-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "blobEndpoint": null, 2021-06-09T14:07:54.530-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "connectionString": null, 2021-06-09T14:07:54.530-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "endpointSuffix": null, 2021-06-09T14:07:54.530-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "name": "accountNameAccountKey", 2021-06-09T14:07:54.531-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "scope": "linksDataverse", 2021-06-09T14:07:54.531-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "sharedAccessSignature": null, 2021-06-09T14:07:54.531-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "type": "azureblob" 2021-06-09T14:07:54.531-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> }, 2021-06-09T14:07:54.531-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> { 2021-06-09T14:07:54.532-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountKey": null, 2021-06-09T14:07:54.532-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "accountName": "devstoreaccount1", 2021-06-09T14:07:54.532-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "blobEndpoint": null, 2021-06-09T14:07:54.532-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "connectionString": null, 2021-06-09T14:07:54.533-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "endpointSuffix": null, 2021-06-09T14:07:54.533-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "name": "accountNameSas", 2021-06-09T14:07:54.533-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "scope": "linksDataverse", 2021-06-09T14:07:54.533-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "sharedAccessSignature": "", 2021-06-09T14:07:54.533-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> "type": "azureblob" 2021-06-09T14:07:54.533-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> } 2021-06-09T14:07:54.534-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> ] 2021-06-09T14:07:54.564-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting AzureBlob link... 2021-06-09T14:07:54.564-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --scope Default --delete --name connectionStringAccountNameAccountKeyLink 2021-06-09T14:07:54.993-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Link deleted 2021-06-09T14:07:55.026-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting AzureBlob link... 2021-06-09T14:07:55.027-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --scope Default --delete --name connectionStringAccountNameSasLink 2021-06-09T14:07:55.412-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> SUCCESS: Link deleted 2021-06-09T14:07:55.447-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting AzureBlob link... 2021-06-09T14:07:55.447-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --scope Default --delete --name accountNameAccountKey 2021-06-09T14:07:55.835-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> ERROR: 24006:Link Default.accountNameAccountKey does not exist 2021-06-09T14:07:55.835-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> 2021-06-09T14:07:55.863-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] Deleting AzureBlob link... 2021-06-09T14:07:55.863-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --scope Default --delete --name accountNameSas 2021-06-09T14:07:56.257-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> ERROR: 24006:Link Default.accountNameSas does not exist 2021-06-09T14:07:56.257-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main+] >> 2021-06-09T14:07:56.286-07:00 INFO ClusterAzureBlobStorageExternalDatasetITD [main] +http://172.17.0.1:36193/analytics/service 2021-06-09T14:07:56.500-07:00 INFO ExternalDatasetTestUtils [main] Adding JSON files to the bucket 2021-06-09T14:07:58.237-07:00 INFO ExternalDatasetTestUtils [main] JSON Files added successfully 2021-06-09T14:07:58.238-07:00 INFO ExternalDatasetTestUtils [main] Adding CSV files to the bucket 2021-06-09T14:07:58.807-07:00 INFO ExternalDatasetTestUtils [main] CSV Files added successfully 2021-06-09T14:07:58.808-07:00 INFO ExternalDatasetTestUtils [main] Adding TSV files to the bucket 2021-06-09T14:07:59.450-07:00 INFO ExternalDatasetTestUtils [main] TSV Files added successfully 2021-06-09T14:07:59.451-07:00 INFO ExternalDatasetTestUtils [main] Adding a big JSON file 2021-06-09T14:08:01.105-07:00 INFO ExternalDatasetTestUtils [main] JSON file added successfully 2021-06-09T14:08:01.105-07:00 INFO ExternalDatasetTestUtils [main] Loading 2999 into over-1000-objects 2021-06-09T14:08:35.046-07:00 INFO ExternalDatasetTestUtils [main] Added 2999 files into over-1000-objects successfully 2021-06-09T14:08:35.047-07:00 INFO ExternalDatasetTestUtils [main] Files added successfully 2021-06-09T14:08:35.047-07:00 INFO ExternalDatasetTestUtils [main] Loading fixed data to fixed-data 2021-06-09T14:08:35.370-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 0: external-dataset: common/create-link-api/000] START 2021-06-09T14:08:35.434-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/create-link-api/000 ... 2021-06-09T14:08:35.477-07:00 INFO TestExecutor [main] Executing test file: create-link-api.000.ddl.sqlpp 2021-06-09T14:08:35.524-07:00 INFO TestExecutor [main] Executing test file: create-link-api.001.ddl.sqlpp 2021-06-09T14:08:35.774-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/000/create-link-api.002.adm 2021-06-09T14:08:35.836-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/000/create-link-api.003.adm 2021-06-09T14:08:35.884-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/000/create-link-api.004.adm 2021-06-09T14:08:35.920-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/000/create-link-api.005.adm 2021-06-09T14:08:35.952-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/000/create-link-api.006.adm 2021-06-09T14:08:35.991-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/000/create-link-api.007.adm 2021-06-09T14:08:36.024-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/000/create-link-api.008.adm 2021-06-09T14:08:36.026-07:00 INFO TestExecutor [main] Executing test file: create-link-api.009.ddl.sqlpp 2021-06-09T14:08:36.129-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/create-link-api/000 PASSED 2021-06-09T14:08:36.132-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/create-link-api/000... 2021-06-09T14:08:36.716-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:37.192-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/create-link-api/000 2021-06-09T14:08:37.193-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 0: external-dataset: common/create-link-api/000] SUCCEEDED 2021-06-09T14:08:37.196-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 1: external-dataset/common/create-link-api: 001] START 2021-06-09T14:08:37.196-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/create-link-api/001 ... 2021-06-09T14:08:37.197-07:00 INFO TestExecutor [main] Executing test file: create-link-api.000.ddl.sqlpp 2021-06-09T14:08:37.226-07:00 INFO TestExecutor [main] Executing test file: create-link-api.001.ddl.sqlpp 2021-06-09T14:08:37.335-07:00 ERRO asterix [main] 24088:Required parameter 'type' not supplied 2021-06-09T14:08:37.337-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/create-link-api/001/create-link-api.002.post.http raised an (expected) exception 2021-06-09T14:08:37.337-07:00 INFO TestExecutor [main] Executing test file: create-link-api.003.ddl.sqlpp 2021-06-09T14:08:37.414-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/create-link-api/001 PASSED 2021-06-09T14:08:37.414-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/create-link-api: 001... 2021-06-09T14:08:37.506-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:37.821-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/create-link-api: 001 2021-06-09T14:08:37.821-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 1: external-dataset/common/create-link-api: 001] SUCCEEDED 2021-06-09T14:08:37.822-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 2: external-dataset/azure_blob_storage/create-link-api: 002] START 2021-06-09T14:08:37.822-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/002 ... 2021-06-09T14:08:37.823-07:00 INFO TestExecutor [main] Executing test file: create-link-api.000.ddl.sqlpp 2021-06-09T14:08:37.860-07:00 INFO TestExecutor [main] Executing test file: create-link-api.001.ddl.sqlpp 2021-06-09T14:08:37.950-07:00 ERRO asterix [main] 24140:No authentication parameters provided 2021-06-09T14:08:37.951-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/azure_blob_storage/create-link-api/002/create-link-api.002.post.http raised an (expected) exception 2021-06-09T14:08:37.952-07:00 INFO TestExecutor [main] Executing test file: create-link-api.003.ddl.sqlpp 2021-06-09T14:08:38.030-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/002 PASSED 2021-06-09T14:08:38.030-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api: 002... 2021-06-09T14:08:38.095-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:38.378-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api: 002 2021-06-09T14:08:38.379-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 2: external-dataset/azure_blob_storage/create-link-api: 002] SUCCEEDED 2021-06-09T14:08:38.380-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 3: external-dataset/azure_blob_storage/create-link-api: 003] START 2021-06-09T14:08:38.380-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/003 ... 2021-06-09T14:08:38.381-07:00 INFO TestExecutor [main] Executing test file: create-link-api.000.ddl.sqlpp 2021-06-09T14:08:38.422-07:00 INFO TestExecutor [main] Executing test file: create-link-api.001.ddl.sqlpp 2021-06-09T14:08:38.496-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/003/create-link-api.002.txt 2021-06-09T14:08:38.497-07:00 INFO TestExecutor [main] Executing test file: create-link-api.003.ddl.sqlpp 2021-06-09T14:08:38.564-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/003 PASSED 2021-06-09T14:08:38.565-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api: 003... 2021-06-09T14:08:38.622-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:38.915-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api: 003 2021-06-09T14:08:38.916-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 3: external-dataset/azure_blob_storage/create-link-api: 003] SUCCEEDED 2021-06-09T14:08:38.916-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 4: external-dataset/common/create-dataset: 000] START 2021-06-09T14:08:38.917-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/create-dataset/000 ... 2021-06-09T14:08:38.917-07:00 INFO TestExecutor [main] Executing test file: create-dataset.000.ddl.sqlpp 2021-06-09T14:08:39.157-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/create-dataset/000/create-dataset.001.adm 2021-06-09T14:08:39.220-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/create-dataset/000/create-dataset.002.adm 2021-06-09T14:08:39.222-07:00 INFO TestExecutor [main] Executing test file: create-dataset.003.ddl.sqlpp 2021-06-09T14:08:40.275-07:00 INFO TestExecutor [main] Executing test file: create-dataset.004.ddl.sqlpp 2021-06-09T14:08:40.412-07:00 INFO TestExecutor [main] Executing test file: create-dataset.005.ddl.sqlpp 2021-06-09T14:08:40.668-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/create-dataset/000/create-dataset.006.adm 2021-06-09T14:08:40.669-07:00 INFO TestExecutor [main] Executing test file: create-dataset.007.ddl.sqlpp 2021-06-09T14:08:40.808-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/create-dataset/000 PASSED 2021-06-09T14:08:40.808-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/create-dataset: 000... 2021-06-09T14:08:40.874-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:41.124-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/create-dataset: 000 2021-06-09T14:08:41.124-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 4: external-dataset/common/create-dataset: 000] SUCCEEDED 2021-06-09T14:08:41.125-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 5: external-dataset/common/create-dataset: 001] START 2021-06-09T14:08:41.125-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/create-dataset/001 ... 2021-06-09T14:08:41.126-07:00 INFO TestExecutor [main] Executing test file: create-dataset.000.ddl.sqlpp 2021-06-09T14:08:41.263-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/create-dataset/001/create-dataset.000.ddl.sqlpp raised an (expected) exception 2021-06-09T14:08:41.264-07:00 INFO TestExecutor [main] Executing test file: create-dataset.001.ddl.sqlpp 2021-06-09T14:08:41.387-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/create-dataset/001/create-dataset.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:08:41.388-07:00 INFO TestExecutor [main] Executing test file: create-dataset.002.ddl.sqlpp 2021-06-09T14:08:41.506-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/create-dataset/001 PASSED 2021-06-09T14:08:41.506-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/create-dataset: 001... 2021-06-09T14:08:41.567-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:41.844-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/create-dataset: 001 2021-06-09T14:08:41.844-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 5: external-dataset/common/create-dataset: 001] SUCCEEDED 2021-06-09T14:08:41.845-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 6: external-dataset/common/drop-link: 000] START 2021-06-09T14:08:41.845-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/drop-link/000 ... 2021-06-09T14:08:41.846-07:00 INFO TestExecutor [main] Executing test file: drop-link.000.ddl.sqlpp 2021-06-09T14:08:42.074-07:00 INFO TestExecutor [main] Executing test file: drop-link.001.ddl.sqlpp 2021-06-09T14:08:42.097-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/drop-link/000/drop-link.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:08:42.098-07:00 INFO TestExecutor [main] Executing test file: drop-link.002.ddl.sqlpp 2021-06-09T14:08:42.194-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/drop-link/000/drop-link.003.adm 2021-06-09T14:08:42.195-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/drop-link/000 PASSED 2021-06-09T14:08:42.195-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/drop-link: 000... 2021-06-09T14:08:42.253-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:42.295-07:00 INFO TestExecutor [main] Disconnecting link test.Local 2021-06-09T14:08:42.501-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: test 2021-06-09T14:08:42.559-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/drop-link: 000 2021-06-09T14:08:42.560-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 6: external-dataset/common/drop-link: 000] SUCCEEDED 2021-06-09T14:08:42.564-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 7: external-dataset/common/query-dataset: json/json] START 2021-06-09T14:08:42.564-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/json/json ... 2021-06-09T14:08:42.565-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:42.713-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.001.adm 2021-06-09T14:08:42.717-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:43.506-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.003.adm 2021-06-09T14:08:43.698-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.004.adm 2021-06-09T14:08:43.872-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.005.adm 2021-06-09T14:08:44.067-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.006.adm 2021-06-09T14:08:44.255-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.007.adm 2021-06-09T14:08:44.447-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.008.adm 2021-06-09T14:08:45.126-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/json/external_dataset.009.adm 2021-06-09T14:08:45.128-07:00 INFO TestExecutor [main] Executing test file: query-dataset.099.ddl.sqlpp 2021-06-09T14:08:45.194-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/json/json PASSED 2021-06-09T14:08:45.194-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: json/json... 2021-06-09T14:08:45.248-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:45.289-07:00 INFO TestExecutor [main] Disconnecting link test2.Local 2021-06-09T14:08:45.495-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: test2 2021-06-09T14:08:45.541-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: json/json 2021-06-09T14:08:45.542-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 7: external-dataset/common/query-dataset: json/json] SUCCEEDED 2021-06-09T14:08:45.542-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 8: external-dataset/common/query-dataset: json/gz] START 2021-06-09T14:08:45.542-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/json/gz ... 2021-06-09T14:08:45.544-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:45.703-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/gz/external_dataset.001.adm 2021-06-09T14:08:45.705-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:46.160-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/gz/external_dataset.003.adm 2021-06-09T14:08:46.333-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/gz/external_dataset.004.adm 2021-06-09T14:08:46.499-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/gz/external_dataset.005.adm 2021-06-09T14:08:46.646-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/gz/external_dataset.006.adm 2021-06-09T14:08:46.831-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/gz/external_dataset.007.adm 2021-06-09T14:08:47.011-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/gz/external_dataset.008.adm 2021-06-09T14:08:47.012-07:00 INFO TestExecutor [main] Executing test file: query-dataset.009.ddl.sqlpp 2021-06-09T14:08:47.080-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/json/gz PASSED 2021-06-09T14:08:47.080-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: json/gz... 2021-06-09T14:08:47.138-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:47.168-07:00 INFO TestExecutor [main] Disconnecting link test2.Local 2021-06-09T14:08:47.382-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: test2 2021-06-09T14:08:47.428-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: json/gz 2021-06-09T14:08:47.429-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 8: external-dataset/common/query-dataset: json/gz] SUCCEEDED 2021-06-09T14:08:47.431-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 9: external-dataset/common/query-dataset: json/mixed] START 2021-06-09T14:08:47.431-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/json/mixed ... 2021-06-09T14:08:47.433-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:47.570-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/mixed/external_dataset.001.adm 2021-06-09T14:08:47.571-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:48.101-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/mixed/external_dataset.003.adm 2021-06-09T14:08:48.367-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/mixed/external_dataset.004.adm 2021-06-09T14:08:48.611-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/mixed/external_dataset.005.adm 2021-06-09T14:08:48.903-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/mixed/external_dataset.006.adm 2021-06-09T14:08:49.173-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/mixed/external_dataset.007.adm 2021-06-09T14:08:49.409-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/json/mixed/external_dataset.008.adm 2021-06-09T14:08:49.410-07:00 INFO TestExecutor [main] Executing test file: query-dataset.009.ddl.sqlpp 2021-06-09T14:08:49.463-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/json/mixed PASSED 2021-06-09T14:08:49.463-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: json/mixed... 2021-06-09T14:08:49.523-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:49.546-07:00 INFO TestExecutor [main] Disconnecting link test2.Local 2021-06-09T14:08:49.741-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: test2 2021-06-09T14:08:49.781-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: json/mixed 2021-06-09T14:08:49.781-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 9: external-dataset/common/query-dataset: json/mixed] SUCCEEDED 2021-06-09T14:08:49.782-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 10: external-dataset/common/query-dataset: csv/csv] START 2021-06-09T14:08:49.782-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/csv/csv ... 2021-06-09T14:08:49.782-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:49.865-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/csv/csv/query-dataset.000.adm 2021-06-09T14:08:49.866-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:50.158-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/csv/csv/query-dataset.003.adm 2021-06-09T14:08:50.159-07:00 INFO TestExecutor [main] Executing test file: query-dataset.004.ddl.sqlpp 2021-06-09T14:08:50.212-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/csv/csv PASSED 2021-06-09T14:08:50.212-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: csv/csv... 2021-06-09T14:08:50.261-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:50.448-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: csv/csv 2021-06-09T14:08:50.448-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 10: external-dataset/common/query-dataset: csv/csv] SUCCEEDED 2021-06-09T14:08:50.449-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 11: external-dataset/common/query-dataset: csv/gz] START 2021-06-09T14:08:50.450-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/csv/gz ... 2021-06-09T14:08:50.451-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:50.553-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/csv/gz/query-dataset.000.adm 2021-06-09T14:08:50.554-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:50.843-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/csv/gz/query-dataset.003.adm 2021-06-09T14:08:50.844-07:00 INFO TestExecutor [main] Executing test file: query-dataset.004.ddl.sqlpp 2021-06-09T14:08:50.892-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/csv/gz PASSED 2021-06-09T14:08:50.892-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: csv/gz... 2021-06-09T14:08:50.942-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:51.115-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: csv/gz 2021-06-09T14:08:51.115-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 11: external-dataset/common/query-dataset: csv/gz] SUCCEEDED 2021-06-09T14:08:51.116-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 12: external-dataset/common/query-dataset: csv/mixed] START 2021-06-09T14:08:51.116-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/csv/mixed ... 2021-06-09T14:08:51.118-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:51.204-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/csv/mixed/query-dataset.000.adm 2021-06-09T14:08:51.205-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:51.633-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/csv/mixed/query-dataset.003.adm 2021-06-09T14:08:51.634-07:00 INFO TestExecutor [main] Executing test file: query-dataset.004.ddl.sqlpp 2021-06-09T14:08:51.677-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/csv/mixed PASSED 2021-06-09T14:08:51.678-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: csv/mixed... 2021-06-09T14:08:51.718-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:51.885-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: csv/mixed 2021-06-09T14:08:51.885-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 12: external-dataset/common/query-dataset: csv/mixed] SUCCEEDED 2021-06-09T14:08:51.886-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 13: external-dataset/common/query-dataset: tsv/tsv] START 2021-06-09T14:08:51.886-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/tsv/tsv ... 2021-06-09T14:08:51.888-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:51.981-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/tsv/tsv/query-dataset.000.adm 2021-06-09T14:08:51.982-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:52.248-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/tsv/tsv/query-dataset.003.adm 2021-06-09T14:08:52.250-07:00 INFO TestExecutor [main] Executing test file: query-dataset.004.ddl.sqlpp 2021-06-09T14:08:52.301-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/tsv/tsv PASSED 2021-06-09T14:08:52.301-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: tsv/tsv... 2021-06-09T14:08:52.355-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:52.556-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: tsv/tsv 2021-06-09T14:08:52.556-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 13: external-dataset/common/query-dataset: tsv/tsv] SUCCEEDED 2021-06-09T14:08:52.557-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 14: external-dataset/common/query-dataset: tsv/gz] START 2021-06-09T14:08:52.557-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/tsv/gz ... 2021-06-09T14:08:52.558-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:52.637-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/tsv/gz/query-dataset.000.adm 2021-06-09T14:08:52.638-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:52.930-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/tsv/gz/query-dataset.003.adm 2021-06-09T14:08:52.931-07:00 INFO TestExecutor [main] Executing test file: query-dataset.004.ddl.sqlpp 2021-06-09T14:08:52.978-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/tsv/gz PASSED 2021-06-09T14:08:52.978-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: tsv/gz... 2021-06-09T14:08:53.019-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:53.190-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: tsv/gz 2021-06-09T14:08:53.191-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 14: external-dataset/common/query-dataset: tsv/gz] SUCCEEDED 2021-06-09T14:08:53.193-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 15: external-dataset/common/query-dataset: tsv/mixed] START 2021-06-09T14:08:53.194-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/tsv/mixed ... 2021-06-09T14:08:53.195-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:53.263-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/tsv/mixed/query-dataset.000.adm 2021-06-09T14:08:53.264-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:53.740-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/tsv/mixed/query-dataset.003.adm 2021-06-09T14:08:53.742-07:00 INFO TestExecutor [main] Executing test file: query-dataset.004.ddl.sqlpp 2021-06-09T14:08:53.795-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/tsv/mixed PASSED 2021-06-09T14:08:53.795-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: tsv/mixed... 2021-06-09T14:08:53.854-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:54.047-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: tsv/mixed 2021-06-09T14:08:54.047-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 15: external-dataset/common/query-dataset: tsv/mixed] SUCCEEDED 2021-06-09T14:08:54.048-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 16: external-dataset/common/query-dataset: empty-string-definition] START 2021-06-09T14:08:54.049-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/empty-string-definition ... 2021-06-09T14:08:54.051-07:00 INFO TestExecutor [main] Executing test file: query-dataset.000.ddl.sqlpp 2021-06-09T14:08:54.150-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/empty-string-definition/external_dataset.001.adm 2021-06-09T14:08:54.151-07:00 INFO TestExecutor [main] Executing test file: query-dataset.002.ddl.sqlpp 2021-06-09T14:08:54.332-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/empty-string-definition/external_dataset.003.adm 2021-06-09T14:08:54.333-07:00 INFO TestExecutor [main] Executing test file: query-dataset.099.ddl.sqlpp 2021-06-09T14:08:54.371-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/empty-string-definition PASSED 2021-06-09T14:08:54.371-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: empty-string-definition... 2021-06-09T14:08:54.415-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:08:54.568-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: empty-string-definition 2021-06-09T14:08:54.569-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 16: external-dataset/common/query-dataset: empty-string-definition] SUCCEEDED 2021-06-09T14:08:54.569-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 17: external-dataset/common/query-dataset: over-1000-objects] START 2021-06-09T14:08:54.570-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/over-1000-objects ... 2021-06-09T14:08:54.570-07:00 INFO TestExecutor [main] Executing test file: over-1000-objects.000.ddl.sqlpp 2021-06-09T14:08:54.645-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/over-1000-objects/over-1000-objects.001.adm 2021-06-09T14:08:54.646-07:00 INFO TestExecutor [main] Executing test file: over-1000-objects.002.ddl.sqlpp 2021-06-09T14:09:31.256-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/over-1000-objects/over-1000-objects.003.adm 2021-06-09T14:09:31.256-07:00 INFO TestExecutor [main] Executing test file: over-1000-objects.099.ddl.sqlpp 2021-06-09T14:09:31.304-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/over-1000-objects PASSED 2021-06-09T14:09:31.305-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: over-1000-objects... 2021-06-09T14:09:31.338-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:31.467-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: over-1000-objects 2021-06-09T14:09:31.467-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 17: external-dataset/common/query-dataset: over-1000-objects] SUCCEEDED 2021-06-09T14:09:31.468-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 18: external-dataset/azure_blob_storage/alter-drop-link: alter-to-invalid-link-endpoint] START 2021-06-09T14:09:31.468-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/alter-drop-link/alter-to-invalid-link-endpoint ... 2021-06-09T14:09:31.470-07:00 INFO TestExecutor [main] Executing test file: alter-to-invalid-link-endpoint.000.ddl.sqlpp 2021-06-09T14:09:31.516-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/alter-drop-link/alter-to-invalid-link-endpoint/alter-to-invalid-link-endpoint.001.adm 2021-06-09T14:09:31.517-07:00 INFO TestExecutor [main] Executing test file: alter-to-invalid-link-endpoint.002.ddl.sqlpp 2021-06-09T14:09:31.633-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/alter-drop-link/alter-to-invalid-link-endpoint/alter-to-invalid-link-endpoint.003.adm 2021-06-09T14:09:31.656-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/alter-drop-link/alter-to-invalid-link-endpoint/alter-to-invalid-link-endpoint.004.adm 2021-06-09T14:09:31.694-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/azure_blob_storage/alter-drop-link/alter-to-invalid-link-endpoint/alter-to-invalid-link-endpoint.005.query.sqlpp raised an (expected) exception 2021-06-09T14:09:31.694-07:00 INFO TestExecutor [main] Executing test file: alter-to-invalid-link-endpoint.006.ddl.sqlpp 2021-06-09T14:09:31.734-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/alter-drop-link/alter-to-invalid-link-endpoint/alter-to-invalid-link-endpoint.006.adm 2021-06-09T14:09:31.734-07:00 INFO TestExecutor [main] Executing test file: alter-to-invalid-link-endpoint.999.ddl.sqlpp 2021-06-09T14:09:31.777-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/alter-drop-link/alter-to-invalid-link-endpoint PASSED 2021-06-09T14:09:31.777-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/alter-drop-link: alter-to-invalid-link-endpoint... 2021-06-09T14:09:31.810-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:31.954-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/alter-drop-link: alter-to-invalid-link-endpoint 2021-06-09T14:09:31.954-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 18: external-dataset/azure_blob_storage/alter-drop-link: alter-to-invalid-link-endpoint] SUCCEEDED 2021-06-09T14:09:31.955-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 19: external-dataset/common/alter-drop-link: drop-link-has-datasets] START 2021-06-09T14:09:31.955-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/alter-drop-link/drop-link-has-datasets ... 2021-06-09T14:09:31.956-07:00 INFO TestExecutor [main] Executing test file: drop-link-has-datasets.000.ddl.sqlpp 2021-06-09T14:09:32.012-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/alter-drop-link/drop-link-has-datasets/drop-link-has-datasets.001.adm 2021-06-09T14:09:32.012-07:00 INFO TestExecutor [main] Executing test file: drop-link-has-datasets.002.ddl.sqlpp 2021-06-09T14:09:32.114-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/alter-drop-link/drop-link-has-datasets/drop-link-has-datasets.003.adm 2021-06-09T14:09:32.128-07:00 ERRO asterix [main] 23004:Link test.myLink cannot be dropped because it has the following analytics collections [test.test] 2021-06-09T14:09:32.128-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/alter-drop-link/drop-link-has-datasets/drop-link-has-datasets.004.delete.http raised an (expected) exception 2021-06-09T14:09:32.129-07:00 INFO TestExecutor [main] Executing test file: drop-link-has-datasets.005.ddl.sqlpp 2021-06-09T14:09:32.163-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/alter-drop-link/drop-link-has-datasets/drop-link-has-datasets.006.adm 2021-06-09T14:09:32.164-07:00 INFO TestExecutor [main] Executing test file: drop-link-has-datasets.007.ddl.sqlpp 2021-06-09T14:09:32.190-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/alter-drop-link/drop-link-has-datasets PASSED 2021-06-09T14:09:32.191-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/alter-drop-link: drop-link-has-datasets... 2021-06-09T14:09:32.228-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:32.358-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/alter-drop-link: drop-link-has-datasets 2021-06-09T14:09:32.358-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 19: external-dataset/common/alter-drop-link: drop-link-has-datasets] SUCCEEDED 2021-06-09T14:09:32.360-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 20: external-dataset/common/query-dataset: definition-does-not-exist] START 2021-06-09T14:09:32.360-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/query-dataset/definition-does-not-exist ... 2021-06-09T14:09:32.361-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:32.419-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/definition-does-not-exist/result.001.adm 2021-06-09T14:09:32.419-07:00 INFO TestExecutor [main] Executing test file: test.002.ddl.sqlpp 2021-06-09T14:09:32.505-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/query-dataset/definition-does-not-exist/result.003.adm 2021-06-09T14:09:32.506-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:32.545-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/query-dataset/definition-does-not-exist PASSED 2021-06-09T14:09:32.545-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/query-dataset: definition-does-not-exist... 2021-06-09T14:09:32.602-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:32.750-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/query-dataset: definition-does-not-exist 2021-06-09T14:09:32.750-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 20: external-dataset/common/query-dataset: definition-does-not-exist] SUCCEEDED 2021-06-09T14:09:32.750-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 21: external-dataset/common/negative: 000] START 2021-06-09T14:09:32.751-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/negative/000 ... 2021-06-09T14:09:32.751-07:00 INFO TestExecutor [main] Executing test file: negative.000.ddl.sqlpp 2021-06-09T14:09:32.806-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/negative/000/negative.000.adm 2021-06-09T14:09:32.806-07:00 INFO TestExecutor [main] Executing test file: negative.002.ddl.sqlpp 2021-06-09T14:09:32.860-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/negative/000/negative.002.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:32.861-07:00 INFO TestExecutor [main] Executing test file: negative.003.ddl.sqlpp 2021-06-09T14:09:32.910-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/negative/000/negative.003.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:32.911-07:00 INFO TestExecutor [main] Executing test file: negative.004.ddl.sqlpp 2021-06-09T14:09:32.960-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/negative/000/negative.004.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:32.960-07:00 INFO TestExecutor [main] Executing test file: negative.005.ddl.sqlpp 2021-06-09T14:09:33.016-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/negative/000/negative.005.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:33.017-07:00 INFO TestExecutor [main] Executing test file: negative.006.ddl.sqlpp 2021-06-09T14:09:34.139-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/negative/000/negative.007.query.sqlpp raised an (expected) exception 2021-06-09T14:09:34.140-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/negative/000 PASSED 2021-06-09T14:09:34.140-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/negative: 000... 2021-06-09T14:09:34.180-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:34.197-07:00 INFO TestExecutor [main] Disconnecting link test.Local 2021-06-09T14:09:34.369-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: test 2021-06-09T14:09:34.418-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/negative: 000 2021-06-09T14:09:34.418-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 21: external-dataset/common/negative: 000] SUCCEEDED 2021-06-09T14:09:34.419-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 22: external-dataset/common/negative: create-index] START 2021-06-09T14:09:34.419-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/negative/create-index ... 2021-06-09T14:09:34.420-07:00 INFO TestExecutor [main] Executing test file: create-index.000.ddl.sqlpp 2021-06-09T14:09:34.480-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/negative/000/negative.000.adm 2021-06-09T14:09:34.481-07:00 INFO TestExecutor [main] Executing test file: create-index.002.ddl.sqlpp 2021-06-09T14:09:34.529-07:00 INFO TestExecutor [main] Executing test file: create-index.003.ddl.sqlpp 2021-06-09T14:09:34.556-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/negative/create-index/create-index.003.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:34.557-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/negative/create-index PASSED 2021-06-09T14:09:34.557-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common/negative: create-index... 2021-06-09T14:09:34.590-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:34.610-07:00 INFO TestExecutor [main] Disconnecting link test.Local 2021-06-09T14:09:34.746-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: test 2021-06-09T14:09:34.781-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common/negative: create-index 2021-06-09T14:09:34.781-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 22: external-dataset/common/negative: create-index] SUCCEEDED 2021-06-09T14:09:34.782-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 23: external-dataset/common: reserved-link-name] START 2021-06-09T14:09:34.782-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/reserved-link-name ... 2021-06-09T14:09:34.799-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/reserved-link-name/reserved-link-name.000.txt 2021-06-09T14:09:34.800-07:00 INFO TestExecutor [main] Executing test file: reserved-link-name.001.ddl.sqlpp 2021-06-09T14:09:34.817-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/reserved-link-name/reserved-link-name.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:34.817-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/reserved-link-name PASSED 2021-06-09T14:09:34.817-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common: reserved-link-name... 2021-06-09T14:09:34.851-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:34.995-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common: reserved-link-name 2021-06-09T14:09:34.996-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 23: external-dataset/common: reserved-link-name] SUCCEEDED 2021-06-09T14:09:34.996-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 24: external-dataset/parameters-validation: invalid-parameters] START 2021-06-09T14:09:34.996-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/parameters-validation/invalid-parameters ... 2021-06-09T14:09:34.997-07:00 INFO TestExecutor [main] Executing test file: invalid-parameters.001.ddl.sqlpp 2021-06-09T14:09:35.058-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/parameters-validation/invalid-parameters/invalid-parameters.001.adm 2021-06-09T14:09:35.059-07:00 INFO TestExecutor [main] Executing test file: invalid-parameters.003.ddl.sqlpp 2021-06-09T14:09:35.090-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/parameters-validation/invalid-parameters/invalid-parameters.003.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:35.091-07:00 INFO TestExecutor [main] Executing test file: invalid-parameters.004.ddl.sqlpp 2021-06-09T14:09:35.133-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/parameters-validation/invalid-parameters/invalid-parameters.004.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:35.134-07:00 INFO TestExecutor [main] Executing test file: invalid-parameters.005.ddl.sqlpp 2021-06-09T14:09:35.188-07:00 INFO TestExecutor [main] [TEST]: external-dataset/parameters-validation/invalid-parameters PASSED 2021-06-09T14:09:35.188-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/parameters-validation: invalid-parameters... 2021-06-09T14:09:35.218-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:35.365-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/parameters-validation: invalid-parameters 2021-06-09T14:09:35.365-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 24: external-dataset/parameters-validation: invalid-parameters] SUCCEEDED 2021-06-09T14:09:35.366-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 25: external-dataset/common: invalid-dataset-name] START 2021-06-09T14:09:35.366-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/invalid-dataset-name ... 2021-06-09T14:09:35.391-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-dataset-name/invalid-dataset-name.1.adm 2021-06-09T14:09:35.392-07:00 INFO TestExecutor [main] Executing test file: invalid-dataset-name.2.ddl.sqlpp 2021-06-09T14:09:35.409-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataset-name/invalid-dataset-name.2.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:35.410-07:00 INFO TestExecutor [main] Executing test file: invalid-dataset-name.3.ddl.sqlpp 2021-06-09T14:09:35.424-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataset-name/invalid-dataset-name.3.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:35.425-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/invalid-dataset-name PASSED 2021-06-09T14:09:35.427-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common: invalid-dataset-name... 2021-06-09T14:09:35.459-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:35.618-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common: invalid-dataset-name 2021-06-09T14:09:35.618-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 25: external-dataset/common: invalid-dataset-name] SUCCEEDED 2021-06-09T14:09:35.619-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 26: external-dataset/common: invalid-dataverse] START 2021-06-09T14:09:35.619-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/invalid-dataverse ... 2021-06-09T14:09:35.635-07:00 ERRO asterix [main] 24034:Cannot find analytics scope with name fakeDataverse 2021-06-09T14:09:35.635-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataverse/invalid-dataverse.000.post.http raised an (expected) exception 2021-06-09T14:09:35.660-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-dataverse/invalid-dataverse.001.adm 2021-06-09T14:09:35.660-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.002.ddl.sqlpp 2021-06-09T14:09:35.680-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataverse/invalid-dataverse.002.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:35.680-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.003.ddl.sqlpp 2021-06-09T14:09:35.706-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.004.ddl.sqlpp 2021-06-09T14:09:35.731-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataverse/invalid-dataverse.004.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:35.732-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.005.ddl.sqlpp 2021-06-09T14:09:35.757-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataverse/invalid-dataverse.005.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:35.758-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.006.ddl.sqlpp 2021-06-09T14:09:35.783-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.007.ddl.sqlpp 2021-06-09T14:09:35.893-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-dataverse/invalid-dataverse.008.adm 2021-06-09T14:09:35.981-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-dataverse/invalid-dataverse.009.adm 2021-06-09T14:09:35.982-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.010.ddl.sqlpp 2021-06-09T14:09:36.006-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataverse/invalid-dataverse.010.ddl.sqlpp issued an (expected) warning 2021-06-09T14:09:36.007-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.011.ddl.sqlpp 2021-06-09T14:09:36.026-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.012.ddl.sqlpp 2021-06-09T14:09:36.061-07:00 ERRO asterix [main] 24034:Cannot find analytics scope with name fakeDataverse 2021-06-09T14:09:36.062-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataverse/invalid-dataverse.013.delete.http raised an (expected) exception 2021-06-09T14:09:36.081-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-dataverse/invalid-dataverse.014.adm 2021-06-09T14:09:36.082-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.015.ddl.sqlpp 2021-06-09T14:09:36.106-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-dataverse/invalid-dataverse.015.ddl.sqlpp issued an (expected) warning 2021-06-09T14:09:36.107-07:00 INFO TestExecutor [main] Executing test file: invalid-dataverse.016.ddl.sqlpp 2021-06-09T14:09:36.127-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/invalid-dataverse PASSED 2021-06-09T14:09:36.127-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common: invalid-dataverse... 2021-06-09T14:09:36.173-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:36.343-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common: invalid-dataverse 2021-06-09T14:09:36.344-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 26: external-dataset/common: invalid-dataverse] SUCCEEDED 2021-06-09T14:09:36.344-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 27: external-dataset/common: invalid-schema/schema-for-json] START 2021-06-09T14:09:36.344-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/invalid-schema/schema-for-json ... 2021-06-09T14:09:36.345-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:36.407-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-schema/schema-for-json/result.000.adm 2021-06-09T14:09:36.408-07:00 INFO TestExecutor [main] Executing test file: test.003.ddl.sqlpp 2021-06-09T14:09:36.429-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-schema/schema-for-json/test.003.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:36.430-07:00 INFO TestExecutor [main] Executing test file: test.004.ddl.sqlpp 2021-06-09T14:09:36.460-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-schema/schema-for-json/test.004.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:36.460-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/invalid-schema/schema-for-json PASSED 2021-06-09T14:09:36.460-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common: invalid-schema/schema-for-json... 2021-06-09T14:09:36.496-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:36.516-07:00 INFO TestExecutor [main] Disconnecting link test.Local 2021-06-09T14:09:36.656-07:00 INFO TestExecutor [main] Last test left some garbage. Dropping dataverses: test 2021-06-09T14:09:36.690-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common: invalid-schema/schema-for-json 2021-06-09T14:09:36.691-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 27: external-dataset/common: invalid-schema/schema-for-json] SUCCEEDED 2021-06-09T14:09:36.691-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 28: external-dataset/common: invalid-schema/no-schema-for-csv] START 2021-06-09T14:09:36.691-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/invalid-schema/no-schema-for-csv ... 2021-06-09T14:09:36.940-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-schema/no-schema-for-csv/result.000.adm 2021-06-09T14:09:36.941-07:00 INFO TestExecutor [main] Executing test file: test.002.ddl.sqlpp 2021-06-09T14:09:36.969-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-schema/no-schema-for-csv/test.002.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:36.970-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/invalid-schema/no-schema-for-csv PASSED 2021-06-09T14:09:36.970-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common: invalid-schema/no-schema-for-csv... 2021-06-09T14:09:37.002-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:37.158-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common: invalid-schema/no-schema-for-csv 2021-06-09T14:09:37.159-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 28: external-dataset/common: invalid-schema/no-schema-for-csv] SUCCEEDED 2021-06-09T14:09:37.159-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 29: external-dataset/common: invalid-schema/no-schema-for-tsv] START 2021-06-09T14:09:37.159-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/invalid-schema/no-schema-for-tsv ... 2021-06-09T14:09:37.182-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/invalid-schema/no-schema-for-tsv/result.000.adm 2021-06-09T14:09:37.183-07:00 INFO TestExecutor [main] Executing test file: test.002.ddl.sqlpp 2021-06-09T14:09:37.215-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/invalid-schema/no-schema-for-tsv/test.002.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:37.216-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/invalid-schema/no-schema-for-tsv PASSED 2021-06-09T14:09:37.216-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common: invalid-schema/no-schema-for-tsv... 2021-06-09T14:09:37.258-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:37.446-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common: invalid-schema/no-schema-for-tsv 2021-06-09T14:09:37.446-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 29: external-dataset/common: invalid-schema/no-schema-for-tsv] SUCCEEDED 2021-06-09T14:09:37.447-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 30: external-dataset/common: bucket-does-not-exist] START 2021-06-09T14:09:37.447-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/bucket-does-not-exist ... 2021-06-09T14:09:37.448-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:37.531-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/bucket-does-not-exist/result.001.adm 2021-06-09T14:09:37.532-07:00 INFO TestExecutor [main] Executing test file: test.002.ddl.sqlpp 2021-06-09T14:09:37.615-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/bucket-does-not-exist/test.002.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:37.616-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:37.661-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/bucket-does-not-exist PASSED 2021-06-09T14:09:37.661-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/common: bucket-does-not-exist... 2021-06-09T14:09:37.701-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:37.962-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/common: bucket-does-not-exist 2021-06-09T14:09:37.963-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 30: external-dataset/common: bucket-does-not-exist] SUCCEEDED 2021-06-09T14:09:37.964-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 31: external-dataset: common/no-files-returned/definition-points-to-nothing] START 2021-06-09T14:09:37.964-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/no-files-returned/definition-points-to-nothing ... 2021-06-09T14:09:37.986-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/no-files-returned/definition-points-to-nothing/result.000.adm 2021-06-09T14:09:37.988-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:38.014-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/no-files-returned/definition-points-to-nothing/test.001.ddl.sqlpp issued an (expected) warning 2021-06-09T14:09:38.061-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/no-files-returned/definition-points-to-nothing/result.002.adm 2021-06-09T14:09:38.062-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/no-files-returned/definition-points-to-nothing/test.002.query.sqlpp issued an (expected) warning 2021-06-09T14:09:38.062-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/no-files-returned/definition-points-to-nothing PASSED 2021-06-09T14:09:38.062-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/no-files-returned/definition-points-to-nothing... 2021-06-09T14:09:38.113-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:38.197-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:38.307-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/no-files-returned/definition-points-to-nothing 2021-06-09T14:09:38.307-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 31: external-dataset: common/no-files-returned/definition-points-to-nothing] SUCCEEDED 2021-06-09T14:09:38.308-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 32: external-dataset: common/no-files-returned/exclude-all-files] START 2021-06-09T14:09:38.308-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/no-files-returned/exclude-all-files ... 2021-06-09T14:09:38.331-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/no-files-returned/exclude-all-files/result.000.adm 2021-06-09T14:09:38.332-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:38.447-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/no-files-returned/exclude-all-files/result.002.adm 2021-06-09T14:09:38.448-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/no-files-returned/exclude-all-files/test.002.query.sqlpp issued an (expected) warning 2021-06-09T14:09:38.448-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/no-files-returned/exclude-all-files PASSED 2021-06-09T14:09:38.448-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/no-files-returned/exclude-all-files... 2021-06-09T14:09:38.483-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:38.566-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:38.678-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/no-files-returned/exclude-all-files 2021-06-09T14:09:38.678-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 32: external-dataset: common/no-files-returned/exclude-all-files] SUCCEEDED 2021-06-09T14:09:38.680-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 33: external-dataset: common/no-files-returned/include-no-files] START 2021-06-09T14:09:38.680-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/no-files-returned/include-no-files ... 2021-06-09T14:09:38.704-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/no-files-returned/include-no-files/result.000.adm 2021-06-09T14:09:38.705-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:38.795-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/no-files-returned/include-no-files/result.002.adm 2021-06-09T14:09:38.796-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/no-files-returned/include-no-files/test.002.query.sqlpp issued an (expected) warning 2021-06-09T14:09:38.796-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/no-files-returned/include-no-files PASSED 2021-06-09T14:09:38.796-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/no-files-returned/include-no-files... 2021-06-09T14:09:38.832-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:38.905-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:39.008-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/no-files-returned/include-no-files 2021-06-09T14:09:39.008-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 33: external-dataset: common/no-files-returned/include-no-files] SUCCEEDED 2021-06-09T14:09:39.010-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 34: external-dataset: common/include-exclude/bad-name-1] START 2021-06-09T14:09:39.010-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/bad-name-1 ... 2021-06-09T14:09:39.042-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/bad-name-1/result.000.adm 2021-06-09T14:09:39.043-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:39.078-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/include-exclude/bad-name-1/test.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:39.079-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:39.092-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/bad-name-1 PASSED 2021-06-09T14:09:39.093-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/bad-name-1... 2021-06-09T14:09:39.128-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:39.289-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/bad-name-1 2021-06-09T14:09:39.290-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 34: external-dataset: common/include-exclude/bad-name-1] SUCCEEDED 2021-06-09T14:09:39.293-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 35: external-dataset: common/include-exclude/bad-name-2] START 2021-06-09T14:09:39.293-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/bad-name-2 ... 2021-06-09T14:09:39.320-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/bad-name-2/result.000.adm 2021-06-09T14:09:39.322-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:39.360-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/include-exclude/bad-name-2/test.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:39.361-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:39.393-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/bad-name-2 PASSED 2021-06-09T14:09:39.394-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/bad-name-2... 2021-06-09T14:09:39.443-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:39.609-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/bad-name-2 2021-06-09T14:09:39.609-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 35: external-dataset: common/include-exclude/bad-name-2] SUCCEEDED 2021-06-09T14:09:39.610-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 36: external-dataset: common/include-exclude/bad-name-3] START 2021-06-09T14:09:39.610-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/bad-name-3 ... 2021-06-09T14:09:39.636-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/bad-name-3/result.000.adm 2021-06-09T14:09:39.637-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:39.664-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/include-exclude/bad-name-3/test.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:39.665-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:39.678-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/bad-name-3 PASSED 2021-06-09T14:09:39.678-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/bad-name-3... 2021-06-09T14:09:39.706-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:39.855-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/bad-name-3 2021-06-09T14:09:39.855-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 36: external-dataset: common/include-exclude/bad-name-3] SUCCEEDED 2021-06-09T14:09:39.856-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 37: external-dataset: common/include-exclude/bad-name-4] START 2021-06-09T14:09:39.856-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/bad-name-4 ... 2021-06-09T14:09:39.878-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/bad-name-4/result.000.adm 2021-06-09T14:09:39.879-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:39.908-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/include-exclude/bad-name-4/test.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:39.909-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:39.923-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/bad-name-4 PASSED 2021-06-09T14:09:39.924-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/bad-name-4... 2021-06-09T14:09:39.954-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:40.108-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/bad-name-4 2021-06-09T14:09:40.109-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 37: external-dataset: common/include-exclude/bad-name-4] SUCCEEDED 2021-06-09T14:09:40.109-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 38: external-dataset: common/include-exclude/non-string] START 2021-06-09T14:09:40.110-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/non-string ... 2021-06-09T14:09:40.137-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/non-string/result.000.adm 2021-06-09T14:09:40.139-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:40.173-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/include-exclude/non-string/test.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:40.174-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:40.192-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/non-string PASSED 2021-06-09T14:09:40.193-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/non-string... 2021-06-09T14:09:40.228-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:40.398-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/non-string 2021-06-09T14:09:40.399-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 38: external-dataset: common/include-exclude/non-string] SUCCEEDED 2021-06-09T14:09:40.400-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 39: external-dataset: common/include-exclude/both] START 2021-06-09T14:09:40.400-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/both ... 2021-06-09T14:09:40.426-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/both/result.000.adm 2021-06-09T14:09:40.428-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:40.458-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/include-exclude/both/test.001.ddl.sqlpp raised an (expected) exception 2021-06-09T14:09:40.459-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:40.480-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/both PASSED 2021-06-09T14:09:40.481-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/both... 2021-06-09T14:09:40.542-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:40.744-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/both 2021-06-09T14:09:40.744-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 39: external-dataset: common/include-exclude/both] SUCCEEDED 2021-06-09T14:09:40.744-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 40: external-dataset: common/include-exclude/exclude-all] START 2021-06-09T14:09:40.745-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/exclude-all ... 2021-06-09T14:09:40.769-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-all/result.000.adm 2021-06-09T14:09:40.770-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:40.885-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-all/result.002.adm 2021-06-09T14:09:40.886-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:40.903-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/exclude-all PASSED 2021-06-09T14:09:40.903-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/exclude-all... 2021-06-09T14:09:40.942-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:41.038-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:41.177-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/exclude-all 2021-06-09T14:09:41.177-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 40: external-dataset: common/include-exclude/exclude-all] SUCCEEDED 2021-06-09T14:09:41.177-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 41: external-dataset: common/include-exclude/exclude-1] START 2021-06-09T14:09:41.177-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/exclude-1 ... 2021-06-09T14:09:41.209-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-1/result.000.adm 2021-06-09T14:09:41.210-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:41.466-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-1/result.002.adm 2021-06-09T14:09:41.467-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:41.485-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/exclude-1 PASSED 2021-06-09T14:09:41.485-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/exclude-1... 2021-06-09T14:09:41.524-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:41.623-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:41.766-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/exclude-1 2021-06-09T14:09:41.766-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 41: external-dataset: common/include-exclude/exclude-1] SUCCEEDED 2021-06-09T14:09:41.767-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 42: external-dataset: common/include-exclude/exclude-2] START 2021-06-09T14:09:41.767-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/exclude-2 ... 2021-06-09T14:09:41.797-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-2/result.000.adm 2021-06-09T14:09:41.797-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:41.972-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-2/result.002.adm 2021-06-09T14:09:41.973-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:41.988-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/exclude-2 PASSED 2021-06-09T14:09:41.988-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/exclude-2... 2021-06-09T14:09:42.042-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:42.140-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:42.262-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/exclude-2 2021-06-09T14:09:42.263-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 42: external-dataset: common/include-exclude/exclude-2] SUCCEEDED 2021-06-09T14:09:42.263-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 43: external-dataset: common/include-exclude/exclude-3] START 2021-06-09T14:09:42.264-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/exclude-3 ... 2021-06-09T14:09:42.300-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-3/result.000.adm 2021-06-09T14:09:42.301-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:42.447-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-3/result.002.adm 2021-06-09T14:09:42.448-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:42.468-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/exclude-3 PASSED 2021-06-09T14:09:42.468-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/exclude-3... 2021-06-09T14:09:42.507-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:42.592-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:42.699-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/exclude-3 2021-06-09T14:09:42.699-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 43: external-dataset: common/include-exclude/exclude-3] SUCCEEDED 2021-06-09T14:09:42.700-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 44: external-dataset: common/include-exclude/exclude-4] START 2021-06-09T14:09:42.701-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/exclude-4 ... 2021-06-09T14:09:42.728-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-4/result.000.adm 2021-06-09T14:09:42.729-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:42.876-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-4/result.002.adm 2021-06-09T14:09:42.877-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:42.893-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/exclude-4 PASSED 2021-06-09T14:09:42.893-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/exclude-4... 2021-06-09T14:09:42.934-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:43.027-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:43.161-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/exclude-4 2021-06-09T14:09:43.161-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 44: external-dataset: common/include-exclude/exclude-4] SUCCEEDED 2021-06-09T14:09:43.162-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 45: external-dataset: common/include-exclude/exclude-5] START 2021-06-09T14:09:43.162-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/exclude-5 ... 2021-06-09T14:09:43.203-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-5/result.000.adm 2021-06-09T14:09:43.204-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:43.459-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-5/result.002.adm 2021-06-09T14:09:43.460-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:43.475-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/exclude-5 PASSED 2021-06-09T14:09:43.475-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/exclude-5... 2021-06-09T14:09:43.512-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:43.598-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:43.735-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/exclude-5 2021-06-09T14:09:43.735-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 45: external-dataset: common/include-exclude/exclude-5] SUCCEEDED 2021-06-09T14:09:43.736-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 46: external-dataset: common/include-exclude/exclude-6] START 2021-06-09T14:09:43.736-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/exclude-6 ... 2021-06-09T14:09:43.764-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-6/result.000.adm 2021-06-09T14:09:43.765-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:43.989-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/exclude-6/result.002.adm 2021-06-09T14:09:43.989-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:44.007-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/exclude-6 PASSED 2021-06-09T14:09:44.007-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/exclude-6... 2021-06-09T14:09:44.046-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:44.136-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:44.248-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/exclude-6 2021-06-09T14:09:44.248-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 46: external-dataset: common/include-exclude/exclude-6] SUCCEEDED 2021-06-09T14:09:44.249-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 47: external-dataset: common/include-exclude/include-all] START 2021-06-09T14:09:44.249-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-all ... 2021-06-09T14:09:44.274-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-all/result.000.adm 2021-06-09T14:09:44.275-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:44.393-07:00 INFO TestExecutor [main] testFile ../cbas-cbserver-test/src/test/resources/runtimets/queries/external-dataset/common/include-exclude/include-all/test.002.query.sqlpp raised an (expected) exception 2021-06-09T14:09:44.394-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:44.409-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-all PASSED 2021-06-09T14:09:44.410-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-all... 2021-06-09T14:09:44.450-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:44.534-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:44.666-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-all 2021-06-09T14:09:44.669-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 47: external-dataset: common/include-exclude/include-all] SUCCEEDED 2021-06-09T14:09:44.673-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 48: external-dataset: common/include-exclude/include-1] START 2021-06-09T14:09:44.673-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-1 ... 2021-06-09T14:09:44.706-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-1/result.000.adm 2021-06-09T14:09:44.707-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:44.819-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-1/result.002.adm 2021-06-09T14:09:44.819-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:44.839-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-1 PASSED 2021-06-09T14:09:44.840-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-1... 2021-06-09T14:09:44.881-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:44.966-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:45.075-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-1 2021-06-09T14:09:45.075-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 48: external-dataset: common/include-exclude/include-1] SUCCEEDED 2021-06-09T14:09:45.075-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 49: external-dataset: common/include-exclude/include-2] START 2021-06-09T14:09:45.076-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-2 ... 2021-06-09T14:09:45.104-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-2/result.000.adm 2021-06-09T14:09:45.104-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:45.222-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-2/result.002.adm 2021-06-09T14:09:45.306-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-2/result.003.adm 2021-06-09T14:09:45.306-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:45.319-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-2 PASSED 2021-06-09T14:09:45.319-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-2... 2021-06-09T14:09:45.352-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:45.435-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:45.565-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-2 2021-06-09T14:09:45.565-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 49: external-dataset: common/include-exclude/include-2] SUCCEEDED 2021-06-09T14:09:45.566-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 50: external-dataset: common/include-exclude/include-3] START 2021-06-09T14:09:45.566-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-3 ... 2021-06-09T14:09:45.592-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-3/result.000.adm 2021-06-09T14:09:45.593-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:45.722-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-3/result.002.adm 2021-06-09T14:09:45.722-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:45.739-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-3 PASSED 2021-06-09T14:09:45.739-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-3... 2021-06-09T14:09:45.778-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:45.856-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:45.983-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-3 2021-06-09T14:09:45.984-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 50: external-dataset: common/include-exclude/include-3] SUCCEEDED 2021-06-09T14:09:45.984-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 51: external-dataset: common/include-exclude/include-4] START 2021-06-09T14:09:45.984-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-4 ... 2021-06-09T14:09:46.006-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-4/result.000.adm 2021-06-09T14:09:46.007-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:46.130-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-4/result.002.adm 2021-06-09T14:09:46.131-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:46.145-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-4 PASSED 2021-06-09T14:09:46.145-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-4... 2021-06-09T14:09:46.179-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:46.260-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:46.386-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-4 2021-06-09T14:09:46.386-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 51: external-dataset: common/include-exclude/include-4] SUCCEEDED 2021-06-09T14:09:46.387-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 52: external-dataset: common/include-exclude/include-5] START 2021-06-09T14:09:46.387-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-5 ... 2021-06-09T14:09:46.411-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-5/result.000.adm 2021-06-09T14:09:46.411-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:46.527-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-5/result.002.adm 2021-06-09T14:09:46.528-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:46.543-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-5 PASSED 2021-06-09T14:09:46.543-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-5... 2021-06-09T14:09:46.575-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:46.665-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:46.775-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-5 2021-06-09T14:09:46.775-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 52: external-dataset: common/include-exclude/include-5] SUCCEEDED 2021-06-09T14:09:46.776-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 53: external-dataset: common/include-exclude/include-6] START 2021-06-09T14:09:46.776-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-6 ... 2021-06-09T14:09:46.807-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-6/result.000.adm 2021-06-09T14:09:46.807-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:46.939-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-6/result.002.adm 2021-06-09T14:09:46.940-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:46.957-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-6 PASSED 2021-06-09T14:09:46.957-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-6... 2021-06-09T14:09:47.002-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:47.108-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:47.244-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-6 2021-06-09T14:09:47.245-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 53: external-dataset: common/include-exclude/include-6] SUCCEEDED 2021-06-09T14:09:47.246-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 54: external-dataset: common/include-exclude/include-7] START 2021-06-09T14:09:47.246-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-7 ... 2021-06-09T14:09:47.270-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-7/result.000.adm 2021-06-09T14:09:47.271-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:47.420-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-7/result.002.adm 2021-06-09T14:09:47.421-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:47.437-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-7 PASSED 2021-06-09T14:09:47.437-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-7... 2021-06-09T14:09:47.471-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:47.574-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:47.700-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-7 2021-06-09T14:09:47.700-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 54: external-dataset: common/include-exclude/include-7] SUCCEEDED 2021-06-09T14:09:47.701-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 55: external-dataset: common/include-exclude/include-8] START 2021-06-09T14:09:47.701-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-8 ... 2021-06-09T14:09:47.731-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-8/result.000.adm 2021-06-09T14:09:47.732-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:47.898-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-8/result.002.adm 2021-06-09T14:09:47.898-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:47.914-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-8 PASSED 2021-06-09T14:09:47.914-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-8... 2021-06-09T14:09:47.959-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:48.052-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:48.173-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-8 2021-06-09T14:09:48.173-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 55: external-dataset: common/include-exclude/include-8] SUCCEEDED 2021-06-09T14:09:48.174-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 56: external-dataset: common/include-exclude/include-9] START 2021-06-09T14:09:48.175-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-9 ... 2021-06-09T14:09:48.198-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-9/result.000.adm 2021-06-09T14:09:48.200-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:48.358-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-9/result.002.adm 2021-06-09T14:09:48.361-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:48.376-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-9 PASSED 2021-06-09T14:09:48.377-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-9... 2021-06-09T14:09:48.413-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:48.507-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:48.621-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-9 2021-06-09T14:09:48.622-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 56: external-dataset: common/include-exclude/include-9] SUCCEEDED 2021-06-09T14:09:48.622-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 57: external-dataset: common/include-exclude/include-10] START 2021-06-09T14:09:48.622-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-10 ... 2021-06-09T14:09:48.644-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-10/result.000.adm 2021-06-09T14:09:48.645-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:48.728-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-10/result.002.adm 2021-06-09T14:09:48.729-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:48.760-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-10 PASSED 2021-06-09T14:09:48.760-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-10... 2021-06-09T14:09:48.799-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:48.896-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:49.007-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-10 2021-06-09T14:09:49.007-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 57: external-dataset: common/include-exclude/include-10] SUCCEEDED 2021-06-09T14:09:49.008-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 58: external-dataset: common/include-exclude/include-11] START 2021-06-09T14:09:49.008-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-11 ... 2021-06-09T14:09:49.034-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-11/result.000.adm 2021-06-09T14:09:49.035-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:49.166-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-11/result.002.adm 2021-06-09T14:09:49.167-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:49.179-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-11 PASSED 2021-06-09T14:09:49.180-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-11... 2021-06-09T14:09:49.210-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:49.290-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:49.405-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-11 2021-06-09T14:09:49.405-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 58: external-dataset: common/include-exclude/include-11] SUCCEEDED 2021-06-09T14:09:49.407-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 59: external-dataset: common/include-exclude/include-12] START 2021-06-09T14:09:49.407-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/common/include-exclude/include-12 ... 2021-06-09T14:09:49.429-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-12/result.000.adm 2021-06-09T14:09:49.430-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:49.522-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/common/include-exclude/include-12/result.002.adm 2021-06-09T14:09:49.523-07:00 INFO TestExecutor [main] Executing test file: test.099.ddl.sqlpp 2021-06-09T14:09:49.537-07:00 INFO TestExecutor [main] [TEST]: external-dataset/common/include-exclude/include-12 PASSED 2021-06-09T14:09:49.537-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset: common/include-exclude/include-12... 2021-06-09T14:09:49.575-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:49.654-07:00 INFO TestExecutor [main] Dropping dataset `Default`.`test` 2021-06-09T14:09:49.764-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset: common/include-exclude/include-12 2021-06-09T14:09:49.765-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 59: external-dataset: common/include-exclude/include-12] SUCCEEDED 2021-06-09T14:09:49.766-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 60: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] START 2021-06-09T14:09:49.767-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods ... 2021-06-09T14:09:49.768-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:49.787-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:49.832-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.002.adm 2021-06-09T14:09:49.833-07:00 INFO TestExecutor [main] Executing test file: test.003.ddl.sqlpp 2021-06-09T14:09:49.967-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.004.adm 2021-06-09T14:09:49.968-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:50.002-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods PASSED 2021-06-09T14:09:50.003-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods... 2021-06-09T14:09:50.051-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:50.181-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods 2021-06-09T14:09:50.182-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 60: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] SUCCEEDED 2021-06-09T14:09:50.183-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 61: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] START 2021-06-09T14:09:50.183-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods ... 2021-06-09T14:09:50.184-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:50.198-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:50.239-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.002.adm 2021-06-09T14:09:50.240-07:00 INFO TestExecutor [main] Executing test file: test.003.ddl.sqlpp 2021-06-09T14:09:50.376-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.004.adm 2021-06-09T14:09:50.377-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:50.420-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods PASSED 2021-06-09T14:09:50.421-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods... 2021-06-09T14:09:50.453-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:50.583-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods 2021-06-09T14:09:50.584-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 61: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] SUCCEEDED 2021-06-09T14:09:50.585-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 62: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] START 2021-06-09T14:09:50.585-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods ... 2021-06-09T14:09:50.586-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:50.601-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:50.635-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.002.adm 2021-06-09T14:09:50.636-07:00 INFO TestExecutor [main] Executing test file: test.003.ddl.sqlpp 2021-06-09T14:09:50.747-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.004.adm 2021-06-09T14:09:50.748-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:50.784-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods PASSED 2021-06-09T14:09:50.785-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods... 2021-06-09T14:09:50.826-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:50.951-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods 2021-06-09T14:09:50.951-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 62: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] SUCCEEDED 2021-06-09T14:09:50.951-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 63: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] START 2021-06-09T14:09:50.951-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods ... 2021-06-09T14:09:50.952-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:50.964-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:50.998-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.002.adm 2021-06-09T14:09:50.999-07:00 INFO TestExecutor [main] Executing test file: test.003.ddl.sqlpp 2021-06-09T14:09:51.106-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods/result.004.adm 2021-06-09T14:09:51.107-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:51.213-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/valid-auth-methods PASSED 2021-06-09T14:09:51.213-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods... 2021-06-09T14:09:51.240-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:51.381-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods 2021-06-09T14:09:51.382-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 63: external-dataset/azure_blob_storage/create-link-api/auth-methods: valid-auth-methods] SUCCEEDED 2021-06-09T14:09:51.382-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 64: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods] START 2021-06-09T14:09:51.382-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods ... 2021-06-09T14:09:51.383-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:51.400-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:51.455-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods/result.002.adm 2021-06-09T14:09:51.456-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:51.550-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods PASSED 2021-06-09T14:09:51.551-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods... 2021-06-09T14:09:51.586-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:51.726-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods 2021-06-09T14:09:51.727-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 64: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods] SUCCEEDED 2021-06-09T14:09:51.727-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 65: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods] START 2021-06-09T14:09:51.728-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods ... 2021-06-09T14:09:51.728-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:51.740-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:51.773-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods/result.002.adm 2021-06-09T14:09:51.774-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:51.839-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods PASSED 2021-06-09T14:09:51.840-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods... 2021-06-09T14:09:51.875-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:51.983-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods 2021-06-09T14:09:51.984-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 65: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods] SUCCEEDED 2021-06-09T14:09:51.984-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 66: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods] START 2021-06-09T14:09:51.984-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods ... 2021-06-09T14:09:51.985-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:51.998-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:52.028-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods/result.002.adm 2021-06-09T14:09:52.028-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:52.064-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-auth-methods PASSED 2021-06-09T14:09:52.064-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods... 2021-06-09T14:09:52.090-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:52.271-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods 2021-06-09T14:09:52.272-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 66: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-auth-methods] SUCCEEDED 2021-06-09T14:09:52.272-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 67: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-no-auth] START 2021-06-09T14:09:52.272-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-no-auth ... 2021-06-09T14:09:52.273-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:52.285-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:52.311-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-no-auth/result.002.adm 2021-06-09T14:09:52.311-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:52.333-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/invalid-no-auth PASSED 2021-06-09T14:09:52.333-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-no-auth... 2021-06-09T14:09:52.358-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:52.478-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-no-auth 2021-06-09T14:09:52.478-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 67: external-dataset/azure_blob_storage/create-link-api/auth-methods: invalid-no-auth] SUCCEEDED 2021-06-09T14:09:52.479-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 68: external-dataset/azure_blob_storage/create-link-api/auth-methods: links-meta] START 2021-06-09T14:09:52.479-07:00 INFO TestExecutor [main] Starting [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/links-meta ... 2021-06-09T14:09:52.480-07:00 INFO TestExecutor [main] Executing test file: test.000.ddl.sqlpp 2021-06-09T14:09:52.492-07:00 INFO TestExecutor [main] Executing test file: test.001.ddl.sqlpp 2021-06-09T14:09:52.522-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/links-meta/result.002.adm 2021-06-09T14:09:52.539-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/links-meta/result.003.adm 2021-06-09T14:09:52.633-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/links-meta/result.004.adm 2021-06-09T14:09:52.655-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/links-meta/result.005.adm 2021-06-09T14:09:52.679-07:00 INFO TestExecutor [main] Expected results file: ../cbas-cbserver-test/src/test/resources/runtimets/results/external-dataset/azure_blob_storage/create-link-api/auth-methods/links-meta/result.006.adm 2021-06-09T14:09:52.680-07:00 INFO TestExecutor [main] Executing test file: test.999.ddl.sqlpp 2021-06-09T14:09:52.722-07:00 INFO TestExecutor [main] [TEST]: external-dataset/azure_blob_storage/create-link-api/auth-methods/links-meta PASSED 2021-06-09T14:09:52.722-07:00 INFO TestExecutor [main] Cleaning up after test case external-dataset/azure_blob_storage/create-link-api/auth-methods: links-meta... 2021-06-09T14:09:52.753-07:00 INFO TestExecutor [main] Disconnecting link Default.Local 2021-06-09T14:09:52.889-07:00 INFO TestExecutor [main] Clean up completed for test case external-dataset/azure_blob_storage/create-link-api/auth-methods: links-meta 2021-06-09T14:09:52.889-07:00 WARN TestMethodTracer [main] ### test[ClusterAzureBlobStorageExternalDatasetIT 68: external-dataset/azure_blob_storage/create-link-api/auth-methods: links-meta] SUCCEEDED 2021-06-09T14:09:52.896-07:00 INFO DockerTestBase [main] Copying files from docker container c5d85d4db0752: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0002_cbas_thisbuild 2021-06-09T14:09:52.901-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 5aa089d97ea5a: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD/0001_azurite_mcr.microsoft.com_azure-storage_azurite_3.9.0 2021-06-09T14:09:53.009-07:00 WARN DockerTestBase [ForkJoinPool.commonPool-worker-3] Error copying files from docker container: 5aa089d97ea5a com.github.dockerjava.api.exception.NotFoundException: Status 404: Could not find the file /opt/couchbase/var/lib/couchbase/logs/ in container 5aa089d97ea5a5499782c6d903559329e29eb204dc90e6ab06c5c54278d2c074 at com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:62) ~[docker-java-transport-jersey-3.2.8.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608) ~[jersey-client-2.30.1.jar:?] at com.github.dockerjava.jaxrs.JerseyDockerHttpClient.execute(JerseyDockerHttpClient.java:297) ~[docker-java-transport-jersey-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.get(DefaultInvocationBuilder.java:202) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.CopyArchiveFromContainerCmdExec.execute(CopyArchiveFromContainerCmdExec.java:30) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.CopyArchiveFromContainerCmdExec.execute(CopyArchiveFromContainerCmdExec.java:13) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.CopyArchiveFromContainerCmdImpl.exec(CopyArchiveFromContainerCmdImpl.java:70) ~[docker-java-core-3.2.8.jar:?] at com.couchbase.analytics.test.docker.DockerTestBase.copyFilesFromContainer(DockerTestBase.java:1008) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.stopContainer(DockerTestBase.java:875) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.stopContainer(DockerTestBase.java:856) ~[test-classes/:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) [?:?] at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) [?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) [?:?] at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) [?:?] at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:746) [?:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?] at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?] at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?] 2021-06-09T14:09:53.021-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 5aa089d97ea5a 2021-06-09T14:09:53.343-07:00 INFO DockerTestBase [main] Stopping docker container: c5d85d4db0752 2021-06-09T14:09:53.829-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 5aa089d97ea5a 2021-06-09T14:09:53.879-07:00 INFO DockerTestBase [main] Stopped docker container: c5d85d4db0752 2021-06-09T14:09:53.880-07:00 INFO DockerTestBase [main] Removing docker container: c5d85d4db0752 2021-06-09T14:09:53.880-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 5aa089d97ea5a 2021-06-09T14:09:53.895-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 5aa089d97ea5a 2021-06-09T14:09:53.895-07:00 INFO DockerTestBase [main] Removed docker container: c5d85d4db0752 2021-06-09T14:09:53.896-07:00 INFO DockerTestBase [main] Removing docker network: d49f88ec6d1f62995f0c08be9babdd8587f00b53224dee0219795721f6b6ac77 2021-06-09T14:09:53.978-07:00 INFO DockerTestBase [main] Removed docker network: d49f88ec6d1f62995f0c08be9babdd8587f00b53224dee0219795721f6b6ac77 [INFO] Tests run: 69, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 166.886 s - in com.couchbase.analytics.test.docker.external.ClusterAzureBlobStorageExternalDatasetITD [INFO] Running com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD 2021-06-09 14:09:56,235 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:09:56.324-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:09:56.361-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:09:56.361-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:09:56.446-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.6.0 2021-06-09T14:09:57.886-07:00 INFO DockerTestBase [main+] >> enterprise-6.6.0: Pulling from library/couchbase 2021-06-09T14:09:58.259-07:00 INFO DockerTestBase [main+] >> 4bbfd2c87b75: Pulling fs layer 2021-06-09T14:09:58.259-07:00 INFO DockerTestBase [main+] >> d2e110be24e1: Pulling fs layer 2021-06-09T14:09:58.259-07:00 INFO DockerTestBase [main+] >> 889a7173dcfe: Pulling fs layer 2021-06-09T14:09:58.260-07:00 INFO DockerTestBase [main+] >> f8723b0ce09e: Pulling fs layer 2021-06-09T14:09:58.260-07:00 INFO DockerTestBase [main+] >> 5ad9d59e6d21: Pulling fs layer 2021-06-09T14:09:58.260-07:00 INFO DockerTestBase [main+] >> 5b8417d9e9d6: Pulling fs layer 2021-06-09T14:09:58.260-07:00 INFO DockerTestBase [main+] >> 0d0f0e79537b: Pulling fs layer 2021-06-09T14:09:58.260-07:00 INFO DockerTestBase [main+] >> affab6c83570: Pulling fs layer 2021-06-09T14:09:58.260-07:00 INFO DockerTestBase [main+] >> 145f40e0dff8: Pulling fs layer 2021-06-09T14:09:58.261-07:00 INFO DockerTestBase [main+] >> 76793676b0ff: Pulling fs layer 2021-06-09T14:09:58.261-07:00 INFO DockerTestBase [main+] >> 3e93f9ace053: Pulling fs layer 2021-06-09T14:09:58.261-07:00 INFO DockerTestBase [main+] >> 4a19a7abac11: Pulling fs layer 2021-06-09T14:09:58.261-07:00 INFO DockerTestBase [main+] >> 6f6b9451341a: Pulling fs layer 2021-06-09T14:09:58.261-07:00 INFO DockerTestBase [main+] >> ae8978f29d13: Pulling fs layer 2021-06-09T14:09:58.261-07:00 INFO DockerTestBase [main+] >> 5ad9d59e6d21: Waiting 2021-06-09T14:09:58.262-07:00 INFO DockerTestBase [main+] >> 5b8417d9e9d6: Waiting 2021-06-09T14:09:58.262-07:00 INFO DockerTestBase [main+] >> 0d0f0e79537b: Waiting 2021-06-09T14:09:58.262-07:00 INFO DockerTestBase [main+] >> affab6c83570: Waiting 2021-06-09T14:09:58.262-07:00 INFO DockerTestBase [main+] >> 4a19a7abac11: Waiting 2021-06-09T14:09:58.262-07:00 INFO DockerTestBase [main+] >> 6f6b9451341a: Waiting 2021-06-09T14:09:58.262-07:00 INFO DockerTestBase [main+] >> 145f40e0dff8: Waiting 2021-06-09T14:09:58.263-07:00 INFO DockerTestBase [main+] >> 76793676b0ff: Waiting 2021-06-09T14:09:58.263-07:00 INFO DockerTestBase [main+] >> 3e93f9ace053: Waiting 2021-06-09T14:09:58.263-07:00 INFO DockerTestBase [main+] >> ae8978f29d13: Waiting 2021-06-09T14:09:58.263-07:00 INFO DockerTestBase [main+] >> f8723b0ce09e: Waiting 2021-06-09T14:09:58.610-07:00 INFO DockerTestBase [main+] >> 889a7173dcfe: Verifying Checksum 2021-06-09T14:09:58.611-07:00 INFO DockerTestBase [main+] >> 889a7173dcfe: Download complete 2021-06-09T14:09:58.612-07:00 INFO DockerTestBase [main+] >> d2e110be24e1: Verifying Checksum 2021-06-09T14:09:58.612-07:00 INFO DockerTestBase [main+] >> d2e110be24e1: Download complete 2021-06-09T14:09:58.949-07:00 INFO DockerTestBase [main+] >> 5ad9d59e6d21: Verifying Checksum 2021-06-09T14:09:58.950-07:00 INFO DockerTestBase [main+] >> 5ad9d59e6d21: Download complete 2021-06-09T14:09:58.958-07:00 INFO DockerTestBase [main+] >> 4bbfd2c87b75: Verifying Checksum 2021-06-09T14:09:58.959-07:00 INFO DockerTestBase [main+] >> 4bbfd2c87b75: Download complete 2021-06-09T14:09:59.051-07:00 INFO DockerTestBase [main+] >> f8723b0ce09e: Verifying Checksum 2021-06-09T14:09:59.051-07:00 INFO DockerTestBase [main+] >> f8723b0ce09e: Download complete 2021-06-09T14:09:59.285-07:00 INFO DockerTestBase [main+] >> 5b8417d9e9d6: Verifying Checksum 2021-06-09T14:09:59.285-07:00 INFO DockerTestBase [main+] >> 5b8417d9e9d6: Download complete 2021-06-09T14:09:59.390-07:00 INFO DockerTestBase [main+] >> affab6c83570: Verifying Checksum 2021-06-09T14:09:59.391-07:00 INFO DockerTestBase [main+] >> affab6c83570: Download complete 2021-06-09T14:09:59.626-07:00 INFO DockerTestBase [main+] >> 145f40e0dff8: Download complete 2021-06-09T14:09:59.729-07:00 INFO DockerTestBase [main+] >> 76793676b0ff: Verifying Checksum 2021-06-09T14:09:59.729-07:00 INFO DockerTestBase [main+] >> 76793676b0ff: Download complete 2021-06-09T14:09:59.970-07:00 INFO DockerTestBase [main+] >> 3e93f9ace053: Download complete 2021-06-09T14:10:00.097-07:00 INFO DockerTestBase [main+] >> 4a19a7abac11: Verifying Checksum 2021-06-09T14:10:00.098-07:00 INFO DockerTestBase [main+] >> 4a19a7abac11: Download complete 2021-06-09T14:10:00.319-07:00 INFO DockerTestBase [main+] >> 6f6b9451341a: Verifying Checksum 2021-06-09T14:10:00.320-07:00 INFO DockerTestBase [main+] >> 6f6b9451341a: Download complete 2021-06-09T14:10:00.412-07:00 INFO DockerTestBase [main+] >> 4bbfd2c87b75: Pull complete 2021-06-09T14:10:00.437-07:00 INFO DockerTestBase [main+] >> ae8978f29d13: Verifying Checksum 2021-06-09T14:10:00.438-07:00 INFO DockerTestBase [main+] >> ae8978f29d13: Download complete 2021-06-09T14:10:00.557-07:00 INFO DockerTestBase [main+] >> d2e110be24e1: Pull complete 2021-06-09T14:10:00.728-07:00 INFO DockerTestBase [main+] >> 889a7173dcfe: Pull complete 2021-06-09T14:10:01.500-07:00 INFO DockerTestBase [main+] >> f8723b0ce09e: Pull complete 2021-06-09T14:10:01.636-07:00 INFO DockerTestBase [main+] >> 5ad9d59e6d21: Pull complete 2021-06-09T14:10:01.805-07:00 INFO DockerTestBase [main+] >> 5b8417d9e9d6: Pull complete 2021-06-09T14:10:06.116-07:00 INFO DockerTestBase [main+] >> 0d0f0e79537b: Verifying Checksum 2021-06-09T14:10:06.117-07:00 INFO DockerTestBase [main+] >> 0d0f0e79537b: Download complete 2021-06-09T14:10:20.199-07:00 INFO DockerTestBase [main+] >> 0d0f0e79537b: Pull complete 2021-06-09T14:10:20.324-07:00 INFO DockerTestBase [main+] >> affab6c83570: Pull complete 2021-06-09T14:10:20.473-07:00 INFO DockerTestBase [main+] >> 145f40e0dff8: Pull complete 2021-06-09T14:10:20.621-07:00 INFO DockerTestBase [main+] >> 76793676b0ff: Pull complete 2021-06-09T14:10:20.741-07:00 INFO DockerTestBase [main+] >> 3e93f9ace053: Pull complete 2021-06-09T14:10:20.861-07:00 INFO DockerTestBase [main+] >> 4a19a7abac11: Pull complete 2021-06-09T14:10:20.976-07:00 INFO DockerTestBase [main+] >> 6f6b9451341a: Pull complete 2021-06-09T14:10:21.096-07:00 INFO DockerTestBase [main+] >> ae8978f29d13: Pull complete 2021-06-09T14:10:21.104-07:00 INFO DockerTestBase [main+] >> Digest: sha256:bd26854f1bfe7b5cabe25b342356867d58a07d469a8e52f080519eb05818cf81 2021-06-09T14:10:21.108-07:00 INFO DockerTestBase [main+] >> Status: Downloaded newer image for couchbase:enterprise-6.6.0 2021-06-09T14:10:21.111-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.6.0 2021-06-09T14:10:21.122-07:00 WARN TestMethodTracer [main] ### upgradeOnlineVerifyClusterCompat START 2021-06-09T14:10:21.183-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:10:22.590-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-f8fa3322-c54a-4715-8f52-2890b5a7a2d9) 2021-06-09T14:10:22.669-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-f8fa3322-c54a-4715-8f52-2890b5a7a2d9, id = 1abc5ffe33d0d468481fac26d35f9a2ddf19f7ab411562041e03695211dd3000 } 2021-06-09T14:10:22.674-07:00 INFO UpgradeTestBase [pool-1] Creating n2 node ... 2021-06-09T14:10:22.675-07:00 INFO UpgradeTestBase [pool-0] Creating n1 node ... 2021-06-09T14:10:22.681-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n2 to /opt/couchbase/var 2021-06-09T14:10:22.690-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n1 to /opt/couchbase/var 2021-06-09T14:10:22.697-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0002_n1_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-f8fa3322-c54a-4715-8f52-2890b5a7a2d9, id = 1abc5ffe33d0d468481fac26d35f9a2ddf19f7ab411562041e03695211dd3000 }) 2021-06-09T14:10:22.697-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0001_n2_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-f8fa3322-c54a-4715-8f52-2890b5a7a2d9, id = 1abc5ffe33d0d468481fac26d35f9a2ddf19f7ab411562041e03695211dd3000 }) 2021-06-09T14:10:22.732-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8091/tcp -> 172.17.0.1:38801/tcp 2021-06-09T14:10:22.732-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8091/tcp -> 172.17.0.1:39853/tcp 2021-06-09T14:10:22.733-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8095/tcp -> 172.17.0.1:34597/tcp 2021-06-09T14:10:22.733-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8095/tcp -> 172.17.0.1:45042/tcp 2021-06-09T14:10:22.896-07:00 INFO DockerTestBase [pool-1] Starting docker container: 8bb81d2e49f55 2021-06-09T14:10:22.905-07:00 INFO DockerTestBase [pool-0] Starting docker container: c7a59a050a632 2021-06-09T14:10:24.158-07:00 INFO DockerTestBase [pool-1] Started docker container: 8bb81d2e49f55 2021-06-09T14:10:24.219-07:00 INFO DockerTestBase [pool-0] Started docker container: c7a59a050a632 2021-06-09T14:10:24.427-07:00 INFO DockerTestBase [pool-0] Container information for c7a59a050a632: host: n1.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.3, 172.17.0.1:38801->192.168.176.3:8091, 172.17.0.1:34597->192.168.176.3:8095 2021-06-09T14:10:24.431-07:00 INFO DockerTestBase [pool-1] Container information for 8bb81d2e49f55: host: n2.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.2, 172.17.0.1:39853->192.168.176.2:8091, 172.17.0.1:45042->192.168.176.2:8095 2021-06-09T14:10:36.349-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1 (192.168.176.3) 2021-06-09T14:10:36.352-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:10:36.420-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node n2 (192.168.176.2) 2021-06-09T14:10:36.420-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:10:37.463-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:10:37.543-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:10:37.572-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:10:37.573-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:10:37.968-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:10:38.000-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:10:50.423-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.2) with services: [ANALYTICS] 2021-06-09T14:10:50.424-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:10:53.380-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:10:53.402-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:10:53.403-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase 2021-06-09T14:11:00.783-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:11:00.783-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:11:00.830-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:11:00.839-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:11:00.840-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:11:00.841-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:11:00.943-07:00 INFO UpgradeTestBase [main] Creating S3 link... 2021-06-09T14:11:00.944-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.2 -u couchbase -p couchbase --create --dataverse Default --name myLink --type s3 --access-key-id dummyAccessKeyId --secret-access-key dummySecretAccessKey --region us-west-2 --service-endpoint http://localhost:8001 2021-06-09T14:11:01.340-07:00 INFO DockerTestBase [main+] >> SUCCESS: Link created 2021-06-09T14:11:01.374-07:00 INFO UpgradeTestBase [main] S3 link created successfully 2021-06-09T14:11:01.374-07:00 INFO UpgradeTestBase [main] Getting link details before upgrade... 2021-06-09T14:11:01.480-07:00 INFO UpgradeTestBase [main] Link details retrieved successfully 2021-06-09T14:11:01.482-07:00 INFO UpgradeTestBase [main] Upgrading [n1.couchbase.host] (online, rebalance) ... 2021-06-09T14:11:01.484-07:00 INFO UpgradeTestBase [pool-0] Creating n1-new node using 'this build' docker image... 2021-06-09T14:11:01.484-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n1new to /opt/couchbase/var 2021-06-09T14:11:01.485-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:11:01.573-07:00 INFO DockerTestBase [pool-0+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:11:01.575-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:11:01.576-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:11:01.576-07:00 INFO DockerTestBase [pool-0+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:11:01.579-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0003_n1new_thisbuild, network = { name = cbas-docker-test-0001-f8fa3322-c54a-4715-8f52-2890b5a7a2d9, id = 1abc5ffe33d0d468481fac26d35f9a2ddf19f7ab411562041e03695211dd3000 }) 2021-06-09T14:11:01.580-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8091/tcp -> 172.17.0.1:37797/tcp 2021-06-09T14:11:01.580-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8095/tcp -> 172.17.0.1:36250/tcp 2021-06-09T14:11:01.620-07:00 INFO DockerTestBase [pool-0] Starting docker container: 1f21aafd222f3 2021-06-09T14:11:02.377-07:00 INFO DockerTestBase [pool-0] Started docker container: 1f21aafd222f3 2021-06-09T14:11:02.391-07:00 INFO DockerTestBase [pool-0] Container information for 1f21aafd222f3: host: n1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:37797->192.168.176.4:8091, 172.17.0.1:36250->192.168.176.4:8095 2021-06-09T14:11:16.457-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1new (192.168.176.4) 2021-06-09T14:11:16.457-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n1new.couchbase.host 2021-06-09T14:11:17.440-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:11:17.475-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node n1new (192.168.176.4) with services: [DATA, ANALYTICS] 2021-06-09T14:11:17.475-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data,analytics 2021-06-09T14:11:22.207-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T14:11:22.260-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T14:11:22.260-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:11:22.262-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase --server-remove n1.couchbase.host 2021-06-09T14:11:42.428-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:11:42.428-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:11:42.458-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:11:42.464-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:11:42.465-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:11:42.468-07:00 INFO DockerTestBase [main] Copying files from docker container c7a59a050a632: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0002_n1_couchbase_enterprise-6.6.0 2021-06-09T14:11:42.698-07:00 INFO DockerTestBase [main] Stopping docker container: c7a59a050a632 2021-06-09T14:11:43.169-07:00 INFO DockerTestBase [main] Stopped docker container: c7a59a050a632 2021-06-09T14:11:43.169-07:00 INFO DockerTestBase [main] Removing docker container: c7a59a050a632 2021-06-09T14:11:43.186-07:00 INFO DockerTestBase [main] Removed docker container: c7a59a050a632 2021-06-09T14:11:43.187-07:00 INFO UpgradeTestBase [main] Upgrading [n2.couchbase.host] (online, rebalance) ... 2021-06-09T14:11:43.187-07:00 INFO UpgradeTestBase [pool-0] Creating n2-new node using 'this build' docker image... 2021-06-09T14:11:43.188-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n2new to /opt/couchbase/var 2021-06-09T14:11:43.188-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0004_n2new_thisbuild, network = { name = cbas-docker-test-0001-f8fa3322-c54a-4715-8f52-2890b5a7a2d9, id = 1abc5ffe33d0d468481fac26d35f9a2ddf19f7ab411562041e03695211dd3000 }) 2021-06-09T14:11:43.189-07:00 INFO DockerTestBase [pool-0] binding exposed port n2new:8091/tcp -> 172.17.0.1:37075/tcp 2021-06-09T14:11:43.189-07:00 INFO DockerTestBase [pool-0] binding exposed port n2new:8095/tcp -> 172.17.0.1:45731/tcp 2021-06-09T14:11:43.256-07:00 INFO DockerTestBase [pool-0] Starting docker container: 562c3f40a9274 2021-06-09T14:11:44.227-07:00 INFO DockerTestBase [pool-0] Started docker container: 562c3f40a9274 2021-06-09T14:11:44.238-07:00 INFO DockerTestBase [pool-0] Container information for 562c3f40a9274: host: n2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:37075->192.168.176.3:8091, 172.17.0.1:45731->192.168.176.3:8095 2021-06-09T14:11:56.284-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n2new (192.168.176.3) 2021-06-09T14:11:56.285-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2new.couchbase.host 2021-06-09T14:11:57.144-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:11:57.186-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node n2new (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:11:57.186-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://n2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:12:01.632-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T14:12:01.689-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T14:12:01.689-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:12:01.690-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove n2.couchbase.host 2021-06-09T14:12:09.107-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:12:09.107-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:12:09.135-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:12:09.145-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:12:09.146-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:12:09.146-07:00 INFO DockerTestBase [main] Copying files from docker container 8bb81d2e49f55: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0001_n2_couchbase_enterprise-6.6.0 2021-06-09T14:12:09.393-07:00 INFO DockerTestBase [main] Stopping docker container: 8bb81d2e49f55 2021-06-09T14:12:09.800-07:00 INFO DockerTestBase [main] Stopped docker container: 8bb81d2e49f55 2021-06-09T14:12:09.800-07:00 INFO DockerTestBase [main] Removing docker container: 8bb81d2e49f55 2021-06-09T14:12:09.814-07:00 INFO DockerTestBase [main] Removed docker container: 8bb81d2e49f55 2021-06-09T14:12:09.815-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:12:28.786-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T14:12:29.009-07:00 INFO UpgradeTestBase [main] Getting link details after upgrade... 2021-06-09T14:12:29.049-07:00 INFO UpgradeTestBase [main] Link details retrieved successfully 2021-06-09T14:12:29.051-07:00 INFO DockerTestBase [main] Copying files from docker container 562c3f40a9274: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0004_n2new_thisbuild 2021-06-09T14:12:29.051-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 1f21aafd222f3: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0003_n1new_thisbuild 2021-06-09T14:12:29.533-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 1f21aafd222f3 2021-06-09T14:12:29.711-07:00 INFO DockerTestBase [main] Stopping docker container: 562c3f40a9274 2021-06-09T14:12:30.087-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 1f21aafd222f3 2021-06-09T14:12:30.141-07:00 INFO DockerTestBase [main] Stopped docker container: 562c3f40a9274 2021-06-09T14:12:30.144-07:00 INFO DockerTestBase [main] Removing docker container: 562c3f40a9274 2021-06-09T14:12:30.146-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 1f21aafd222f3 2021-06-09T14:12:30.164-07:00 INFO DockerTestBase [main] Removed docker container: 562c3f40a9274 2021-06-09T14:12:30.164-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 1f21aafd222f3 2021-06-09T14:12:30.166-07:00 INFO DockerTestBase [main] Removing docker network: 1abc5ffe33d0d468481fac26d35f9a2ddf19f7ab411562041e03695211dd3000 2021-06-09T14:12:30.232-07:00 INFO DockerTestBase [main] Removed docker network: 1abc5ffe33d0d468481fac26d35f9a2ddf19f7ab411562041e03695211dd3000 2021-06-09T14:12:30.239-07:00 WARN TestMethodTracer [main] ### upgradeOnlineVerifyClusterCompat SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 155.309 s - in com.couchbase.analytics.test.docker.external.S3LinkUpgradeFrom660ITD [INFO] Running com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD 2021-06-09 14:12:32,680 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:12:32.765-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:12:32.803-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:12:32.804-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:12:32.879-07:00 WARN TestMethodTracer [main] ### refreshCcIpAddress START 2021-06-09T14:12:32.923-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:12:34.261-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-a101e48e-80a5-4913-ad79-834ac5cf257f) 2021-06-09T14:12:34.337-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-a101e48e-80a5-4913-ad79-834ac5cf257f, id = b100a2f267afc59a10d8595f414a2dc4f4ac42761a749fbb028890ac42e42772 } 2021-06-09T14:12:34.342-07:00 INFO UpgradeTestBase [pool-0] Creating n1 node using 'this build' docker image... 2021-06-09T14:12:34.357-07:00 INFO UpgradeTestBase [pool-2] Creating n3 node using 'this build' docker image... 2021-06-09T14:12:34.362-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:12:34.362-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n3 to /opt/couchbase/var 2021-06-09T14:12:34.364-07:00 INFO UpgradeTestBase [pool-1] Creating n2 node using 'this build' docker image... 2021-06-09T14:12:34.365-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n2 to /opt/couchbase/var 2021-06-09T14:12:34.374-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:12:34.485-07:00 INFO DockerTestBase [pool-0+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:12:34.488-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:12:34.488-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:12:34.489-07:00 INFO DockerTestBase [pool-0+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:12:34.494-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0001_n1_thisbuild, network = { name = cbas-docker-test-0001-a101e48e-80a5-4913-ad79-834ac5cf257f, id = b100a2f267afc59a10d8595f414a2dc4f4ac42761a749fbb028890ac42e42772 }) 2021-06-09T14:12:34.494-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0003_n3_thisbuild, network = { name = cbas-docker-test-0001-a101e48e-80a5-4913-ad79-834ac5cf257f, id = b100a2f267afc59a10d8595f414a2dc4f4ac42761a749fbb028890ac42e42772 }) 2021-06-09T14:12:34.495-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0002_n2_thisbuild, network = { name = cbas-docker-test-0001-a101e48e-80a5-4913-ad79-834ac5cf257f, id = b100a2f267afc59a10d8595f414a2dc4f4ac42761a749fbb028890ac42e42772 }) 2021-06-09T14:12:34.539-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8091/tcp -> 172.17.0.1:40323/tcp 2021-06-09T14:12:34.539-07:00 INFO DockerTestBase [pool-2] binding exposed port n3:8091/tcp -> 172.17.0.1:34623/tcp 2021-06-09T14:12:34.540-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8095/tcp -> 172.17.0.1:41263/tcp 2021-06-09T14:12:34.540-07:00 INFO DockerTestBase [pool-2] binding exposed port n3:8095/tcp -> 172.17.0.1:44726/tcp 2021-06-09T14:12:34.547-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8091/tcp -> 172.17.0.1:40477/tcp 2021-06-09T14:12:34.548-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8095/tcp -> 172.17.0.1:43787/tcp 2021-06-09T14:12:34.776-07:00 INFO DockerTestBase [pool-2] Starting docker container: 145bbbb47d2fb 2021-06-09T14:12:34.783-07:00 INFO DockerTestBase [pool-1] Starting docker container: fbc69a11ec39d 2021-06-09T14:12:34.786-07:00 INFO DockerTestBase [pool-0] Starting docker container: 405fccec9bf53 2021-06-09T14:12:35.886-07:00 INFO DockerTestBase [pool-1] Started docker container: fbc69a11ec39d 2021-06-09T14:12:36.059-07:00 INFO DockerTestBase [pool-2] Started docker container: 145bbbb47d2fb 2021-06-09T14:12:36.121-07:00 INFO DockerTestBase [pool-0] Started docker container: 405fccec9bf53 2021-06-09T14:12:36.165-07:00 INFO DockerTestBase [pool-0] Container information for 405fccec9bf53: host: n1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:40323->192.168.176.3:8091, 172.17.0.1:41263->192.168.176.3:8095 2021-06-09T14:12:36.167-07:00 INFO DockerTestBase [pool-1] Container information for fbc69a11ec39d: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:40477->192.168.176.2:8091, 172.17.0.1:43787->192.168.176.2:8095 2021-06-09T14:12:36.165-07:00 INFO DockerTestBase [pool-2] Container information for 145bbbb47d2fb: host: n3.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:34623->192.168.176.4:8091, 172.17.0.1:44726->192.168.176.4:8095 2021-06-09T14:12:47.401-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node n2 (192.168.176.2) 2021-06-09T14:12:47.406-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:12:47.555-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1 (192.168.176.3) 2021-06-09T14:12:47.555-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:12:47.561-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node n3 (192.168.176.4) 2021-06-09T14:12:47.562-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n3.couchbase.host 2021-06-09T14:12:48.473-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T14:12:48.661-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:12:48.972-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:12:49.030-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.3) with services: [DATA] 2021-06-09T14:12:49.031-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:12:49.561-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:12:49.595-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.2) with services: [ANALYTICS] 2021-06-09T14:12:49.596-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:12:53.731-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:12:53.765-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:12:53.766-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase 2021-06-09T14:12:58.242-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:12:58.242-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:12:58.283-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:12:58.305-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:12:58.306-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:12:58.313-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:13:04.814-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:13:04.815-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n3.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:13:08.908-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:13:08.971-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:13:08.973-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase 2021-06-09T14:13:17.608-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:13:17.609-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:13:17.638-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:13:17.646-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:13:17.647-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:13:17.647-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:13:17.734-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:40477] 2021-06-09T14:13:17.738-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:40477 2021-06-09T14:13:18.808-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:34623] 2021-06-09T14:13:23.873-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:13:23.876-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:13:24.944-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 63 } 2021-06-09T14:13:24.975-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 63 } 2021-06-09T14:13:25.977-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:13:26.245-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2936 } 2021-06-09T14:13:26.246-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2936 } 2021-06-09T14:13:27.247-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:13:27.616-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5859 } 2021-06-09T14:13:27.616-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 5859 } 2021-06-09T14:13:28.802-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:13:28.803-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:13:28.803-07:00 INFO UpgradeTestBase [main] Recreating 'n2' node (image = null, forceNewIp = true)... 2021-06-09T14:13:28.804-07:00 INFO DockerTestBase [main] Copying files from docker container fbc69a11ec39d: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0002_n2_thisbuild 2021-06-09T14:13:29.180-07:00 INFO DockerTestBase [main] Stopping docker container: fbc69a11ec39d 2021-06-09T14:13:29.682-07:00 INFO DockerTestBase [main] Stopped docker container: fbc69a11ec39d 2021-06-09T14:13:29.683-07:00 INFO DockerTestBase [main] Removing docker container: fbc69a11ec39d 2021-06-09T14:13:29.698-07:00 INFO DockerTestBase [main] Removed docker container: fbc69a11ec39d 2021-06-09T14:13:29.700-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0004_n2_thisbuild, network = { name = cbas-docker-test-0001-a101e48e-80a5-4913-ad79-834ac5cf257f, id = b100a2f267afc59a10d8595f414a2dc4f4ac42761a749fbb028890ac42e42772 }) 2021-06-09T14:13:29.701-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:42916/tcp 2021-06-09T14:13:29.702-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:35317/tcp 2021-06-09T14:13:29.761-07:00 INFO DockerTestBase [main] Starting docker container: cdb494929b02d 2021-06-09T14:13:30.542-07:00 INFO DockerTestBase [main] Started docker container: cdb494929b02d 2021-06-09T14:13:30.552-07:00 INFO DockerTestBase [main] Container information for cdb494929b02d: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.178.0, 172.17.0.1:42916->192.168.178.0:8091, 172.17.0.1:35317->192.168.178.0:8095 2021-06-09T14:13:39.802-07:00 INFO IpAddressRefreshITD [main] waiting to 4 minutes for IPs to refresh in the NC container & cluster to be active 2021-06-09T14:13:39.805-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 240s... 2021-06-09T14:13:39.805-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 240s... 2021-06-09T14:14:39.607-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:34623] 2021-06-09T14:14:39.607-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42916] 2021-06-09T14:14:39.608-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:14:39.608-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T14:14:39.608-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:14:39.608-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T14:14:40.593-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2949 } 2021-06-09T14:14:40.593-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 2949 } 2021-06-09T14:14:40.664-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 2949 } 2021-06-09T14:14:40.664-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 1. ResultSet found : { "$1": 2949 } 2021-06-09T14:14:41.595-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:14:42.008-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2949 } 2021-06-09T14:14:42.008-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2949 } 2021-06-09T14:14:42.056-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 2949 } 2021-06-09T14:14:42.056-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 2. ResultSet found : { "$1": 2949 } 2021-06-09T14:14:43.009-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:14:43.057-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T14:14:43.420-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3702 } 2021-06-09T14:14:43.420-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 3702 } 2021-06-09T14:14:43.532-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 3707 } 2021-06-09T14:14:43.532-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 3. ResultSet found : { "$1": 3707 } 2021-06-09T14:14:44.422-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:14:44.593-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5617 } 2021-06-09T14:14:44.594-07:00 INFO DockerTestBase [main] Failed attempt number: 4. ResultSet found : { "$1": 5617 } 2021-06-09T14:14:44.777-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5806 } 2021-06-09T14:14:44.778-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 4. ResultSet found : { "$1": 5806 } 2021-06-09T14:14:45.595-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:14:45.719-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:14:45.719-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:14:45.779-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T14:14:45.925-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T14:14:45.925-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T14:14:45.926-07:00 INFO DockerTestBase [main] Copying files from docker container 145bbbb47d2fb: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0003_n3_thisbuild 2021-06-09T14:14:46.256-07:00 INFO DockerTestBase [main] Stopping docker container: 145bbbb47d2fb 2021-06-09T14:14:46.680-07:00 INFO DockerTestBase [main] Stopped docker container: 145bbbb47d2fb 2021-06-09T14:14:46.692-07:00 INFO DockerTestBase [main] found 2 occurrences of 'ip address updated on refresh' in analytics_info.log on n3 (192.168.176.4) 2021-06-09T14:14:46.694-07:00 INFO DockerTestBase [main] Copying files from docker container cdb494929b02d: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0004_n2_thisbuild 2021-06-09T14:14:46.694-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 405fccec9bf53: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshCcIpAddress/0001_n1_thisbuild 2021-06-09T14:14:47.200-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 405fccec9bf53 2021-06-09T14:14:47.298-07:00 INFO DockerTestBase [main] Stopping docker container: cdb494929b02d 2021-06-09T14:14:47.770-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 405fccec9bf53 2021-06-09T14:14:47.849-07:00 INFO DockerTestBase [main] Stopped docker container: cdb494929b02d 2021-06-09T14:14:47.854-07:00 INFO DockerTestBase [main] Removing docker container: 405fccec9bf53 2021-06-09T14:14:47.854-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: cdb494929b02d 2021-06-09T14:14:47.855-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 145bbbb47d2fb 2021-06-09T14:14:47.869-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: cdb494929b02d 2021-06-09T14:14:47.869-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 145bbbb47d2fb 2021-06-09T14:14:47.869-07:00 INFO DockerTestBase [main] Removed docker container: 405fccec9bf53 2021-06-09T14:14:47.871-07:00 INFO DockerTestBase [main] Removing docker network: b100a2f267afc59a10d8595f414a2dc4f4ac42761a749fbb028890ac42e42772 2021-06-09T14:14:47.937-07:00 INFO DockerTestBase [main] Removed docker network: b100a2f267afc59a10d8595f414a2dc4f4ac42761a749fbb028890ac42e42772 2021-06-09T14:14:47.944-07:00 WARN TestMethodTracer [main] ### refreshCcIpAddress SUCCEEDED 2021-06-09T14:14:47.945-07:00 WARN TestMethodTracer [main] ### refreshNcIpAddress START 2021-06-09T14:14:47.952-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:14:48.008-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0002-61972aba-ec50-4dea-8232-2e1d19a09922) 2021-06-09T14:14:48.060-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0002-61972aba-ec50-4dea-8232-2e1d19a09922, id = 4c3ab4a9ea48d2cb51c116cb1059819c2dbc143aea33948ae6b681c9dbc73801 } 2021-06-09T14:14:48.061-07:00 INFO UpgradeTestBase [pool-0] Creating n1 node using 'this build' docker image... 2021-06-09T14:14:48.063-07:00 INFO UpgradeTestBase [pool-1] Creating n2 node using 'this build' docker image... 2021-06-09T14:14:48.064-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:14:48.064-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshNcIpAddress/0005_n2_thisbuild, network = { name = cbas-docker-test-0002-61972aba-ec50-4dea-8232-2e1d19a09922, id = 4c3ab4a9ea48d2cb51c116cb1059819c2dbc143aea33948ae6b681c9dbc73801 }) 2021-06-09T14:14:48.076-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8091/tcp -> 172.17.0.1:35760/tcp 2021-06-09T14:14:48.078-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8095/tcp -> 172.17.0.1:40176/tcp 2021-06-09T14:14:48.156-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:14:48.157-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshNcIpAddress/0006_n1_thisbuild, network = { name = cbas-docker-test-0002-61972aba-ec50-4dea-8232-2e1d19a09922, id = 4c3ab4a9ea48d2cb51c116cb1059819c2dbc143aea33948ae6b681c9dbc73801 }) 2021-06-09T14:14:48.158-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8091/tcp -> 172.17.0.1:44523/tcp 2021-06-09T14:14:48.158-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8095/tcp -> 172.17.0.1:43625/tcp 2021-06-09T14:14:48.184-07:00 INFO DockerTestBase [pool-1] Starting docker container: 3c2ee6c07ae1e 2021-06-09T14:14:48.227-07:00 INFO DockerTestBase [pool-0] Starting docker container: d435dd7907889 2021-06-09T14:14:49.329-07:00 INFO DockerTestBase [pool-1] Started docker container: 3c2ee6c07ae1e 2021-06-09T14:14:49.379-07:00 INFO DockerTestBase [pool-1] Container information for 3c2ee6c07ae1e: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:35760->192.168.176.2:8091, 172.17.0.1:40176->192.168.176.2:8095 2021-06-09T14:14:49.433-07:00 INFO DockerTestBase [pool-0] Started docker container: d435dd7907889 2021-06-09T14:14:49.452-07:00 INFO DockerTestBase [pool-0] Container information for d435dd7907889: host: n1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:44523->192.168.176.3:8091, 172.17.0.1:43625->192.168.176.3:8095 2021-06-09T14:15:00.876-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1 (192.168.176.3) 2021-06-09T14:15:00.876-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:15:00.958-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node n2 (192.168.176.2) 2021-06-09T14:15:00.958-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:15:01.875-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:15:02.016-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:15:02.052-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:15:02.053-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:15:02.540-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:15:02.584-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.2) with services: [ANALYTICS] 2021-06-09T14:15:02.584-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:15:06.833-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:15:06.873-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:15:06.874-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase 2021-06-09T14:15:26.583-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:15:26.583-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:15:26.626-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:15:26.643-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:15:26.644-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:15:26.644-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:15:26.937-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:44523] 2021-06-09T14:15:26.938-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:44523 2021-06-09T14:15:27.992-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35760] 2021-06-09T14:15:33.525-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:15:33.527-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:15:34.577-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 4 } 2021-06-09T14:15:34.577-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 4 } 2021-06-09T14:15:35.960-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2936 } 2021-06-09T14:15:35.961-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2936 } 2021-06-09T14:15:36.962-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:15:37.243-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:15:37.243-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:15:37.244-07:00 INFO UpgradeTestBase [main] Recreating 'n2' node (image = null, forceNewIp = true)... 2021-06-09T14:15:37.244-07:00 INFO DockerTestBase [main] Copying files from docker container 3c2ee6c07ae1e: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshNcIpAddress/0005_n2_thisbuild 2021-06-09T14:15:37.614-07:00 INFO DockerTestBase [main] Stopping docker container: 3c2ee6c07ae1e 2021-06-09T14:15:38.075-07:00 INFO DockerTestBase [main] Stopped docker container: 3c2ee6c07ae1e 2021-06-09T14:15:38.076-07:00 INFO DockerTestBase [main] Removing docker container: 3c2ee6c07ae1e 2021-06-09T14:15:38.086-07:00 INFO DockerTestBase [main] Removed docker container: 3c2ee6c07ae1e 2021-06-09T14:15:38.087-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshNcIpAddress/0007_n2_thisbuild, network = { name = cbas-docker-test-0002-61972aba-ec50-4dea-8232-2e1d19a09922, id = 4c3ab4a9ea48d2cb51c116cb1059819c2dbc143aea33948ae6b681c9dbc73801 }) 2021-06-09T14:15:38.088-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:33584/tcp 2021-06-09T14:15:38.088-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:44701/tcp 2021-06-09T14:15:38.152-07:00 INFO DockerTestBase [main] Starting docker container: bd2490ea0dc6c 2021-06-09T14:15:39.002-07:00 INFO DockerTestBase [main] Started docker container: bd2490ea0dc6c 2021-06-09T14:15:39.011-07:00 INFO DockerTestBase [main] Container information for bd2490ea0dc6c: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.178.1, 172.17.0.1:33584->192.168.178.1:8091, 172.17.0.1:44701->192.168.178.1:8095 2021-06-09T14:15:50.056-07:00 INFO IpAddressRefreshITD [main] waiting to 4 minutes for IPs to refresh in the CC container & cluster to be active 2021-06-09T14:15:50.057-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 240s... 2021-06-09T14:15:50.057-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 240s... 2021-06-09T14:16:50.936-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:33584] 2021-06-09T14:16:50.937-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:16:50.937-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:16:50.939-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-7] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:44523] 2021-06-09T14:16:50.940-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Waiting for all beers to be ingested... 2021-06-09T14:16:50.940-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-7] using Accept-Charset: UTF-32 2021-06-09T14:16:52.798-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] ResultSet found : { "$1": 2942 } 2021-06-09T14:16:52.799-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Failed attempt number: 1. ResultSet found : { "$1": 2942 } 2021-06-09T14:16:53.001-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2942 } 2021-06-09T14:16:53.001-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 2942 } 2021-06-09T14:16:54.002-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:16:54.154-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] ResultSet found : { "$1": 4405 } 2021-06-09T14:16:54.154-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Failed attempt number: 2. ResultSet found : { "$1": 4405 } 2021-06-09T14:16:54.599-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 4405 } 2021-06-09T14:16:54.599-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 4405 } 2021-06-09T14:16:55.155-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-7] using Accept-Charset: UTF-16BE 2021-06-09T14:16:55.313-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] ResultSet found : { "$1": 4405 } 2021-06-09T14:16:55.313-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Failed attempt number: 3. ResultSet found : { "$1": 4405 } 2021-06-09T14:16:55.600-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:16:55.745-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 4550 } 2021-06-09T14:16:55.745-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 4550 } 2021-06-09T14:16:56.486-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] ResultSet found : { "$1": 5891 } 2021-06-09T14:16:56.486-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Success 2021-06-09T14:16:56.746-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:16:56.844-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:16:56.845-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:16:56.845-07:00 INFO DockerTestBase [main] Copying files from docker container d435dd7907889: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshNcIpAddress/0006_n1_thisbuild 2021-06-09T14:16:57.709-07:00 INFO DockerTestBase [main] Stopping docker container: d435dd7907889 2021-06-09T14:16:58.212-07:00 INFO DockerTestBase [main] Stopped docker container: d435dd7907889 2021-06-09T14:16:58.214-07:00 INFO DockerTestBase [main] found 2 occurrences of 'ip address updated on refresh' in analytics_info.log on n1 (192.168.176.3) 2021-06-09T14:16:58.215-07:00 INFO DockerTestBase [main] Copying files from docker container bd2490ea0dc6c: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD/refreshNcIpAddress/0007_n2_thisbuild 2021-06-09T14:16:58.909-07:00 INFO DockerTestBase [main] Stopping docker container: bd2490ea0dc6c 2021-06-09T14:16:59.283-07:00 INFO DockerTestBase [main] Stopped docker container: bd2490ea0dc6c 2021-06-09T14:16:59.283-07:00 INFO DockerTestBase [main] Removing docker container: bd2490ea0dc6c 2021-06-09T14:16:59.283-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: d435dd7907889 2021-06-09T14:16:59.294-07:00 INFO DockerTestBase [main] Removed docker container: bd2490ea0dc6c 2021-06-09T14:16:59.295-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: d435dd7907889 2021-06-09T14:16:59.295-07:00 INFO DockerTestBase [main] Removing docker network: 4c3ab4a9ea48d2cb51c116cb1059819c2dbc143aea33948ae6b681c9dbc73801 2021-06-09T14:16:59.346-07:00 INFO DockerTestBase [main] Removed docker network: 4c3ab4a9ea48d2cb51c116cb1059819c2dbc143aea33948ae6b681c9dbc73801 2021-06-09T14:16:59.348-07:00 WARN TestMethodTracer [main] ### refreshNcIpAddress SUCCEEDED [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 268.108 s - in com.couchbase.analytics.test.docker.lifecycle.IpAddressRefreshITD [INFO] Running com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD 2021-06-09 14:17:02,847 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:17:02.936-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:17:02.972-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:17:02.973-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:17:03.103-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:17:03.190-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:17:03.191-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:17:03.192-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:17:03.192-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:17:03.200-07:00 WARN TestMethodTracer [main] ### testQuorumFailover START 2021-06-09T14:17:03.242-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:17:04.482-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-7ea02840-34c3-4057-b2e1-ce26bc9fc9e1) 2021-06-09T14:17:04.611-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-7ea02840-34c3-4057-b2e1-ce26bc9fc9e1, id = 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 } 2021-06-09T14:17:04.616-07:00 INFO UpgradeTestBase [pool-1] Creating n2 node using 'this build' docker image... 2021-06-09T14:17:04.617-07:00 INFO UpgradeTestBase [pool-0] Creating n1 node using 'this build' docker image... 2021-06-09T14:17:04.636-07:00 INFO UpgradeTestBase [pool-3] Creating n4 node using 'this build' docker image... 2021-06-09T14:17:04.634-07:00 INFO UpgradeTestBase [pool-2] Creating n3 node using 'this build' docker image... 2021-06-09T14:17:04.641-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n1 to /opt/couchbase/var 2021-06-09T14:17:04.641-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n3 to /opt/couchbase/var 2021-06-09T14:17:04.644-07:00 INFO UpgradeTestBase [pool-3] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n4 to /opt/couchbase/var 2021-06-09T14:17:04.644-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n2 to /opt/couchbase/var 2021-06-09T14:17:04.650-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0002_n2_thisbuild, network = { name = cbas-docker-test-0001-7ea02840-34c3-4057-b2e1-ce26bc9fc9e1, id = 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 }) 2021-06-09T14:17:04.651-07:00 INFO UpgradeTestBase [pool-4] Creating n5 node using 'this build' docker image... 2021-06-09T14:17:04.655-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0001_n1_thisbuild, network = { name = cbas-docker-test-0001-7ea02840-34c3-4057-b2e1-ce26bc9fc9e1, id = 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 }) 2021-06-09T14:17:04.663-07:00 INFO DockerTestBase [pool-3] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0003_n4_thisbuild, network = { name = cbas-docker-test-0001-7ea02840-34c3-4057-b2e1-ce26bc9fc9e1, id = 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 }) 2021-06-09T14:17:04.669-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0004_n3_thisbuild, network = { name = cbas-docker-test-0001-7ea02840-34c3-4057-b2e1-ce26bc9fc9e1, id = 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 }) 2021-06-09T14:17:04.669-07:00 INFO UpgradeTestBase [pool-4] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0005n5 to /opt/couchbase/var 2021-06-09T14:17:04.693-07:00 INFO DockerTestBase [pool-4] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0005_n5_thisbuild, network = { name = cbas-docker-test-0001-7ea02840-34c3-4057-b2e1-ce26bc9fc9e1, id = 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 }) 2021-06-09T14:17:04.719-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8091/tcp -> 172.17.0.1:38319/tcp 2021-06-09T14:17:04.719-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8091/tcp -> 172.17.0.1:45949/tcp 2021-06-09T14:17:04.720-07:00 INFO DockerTestBase [pool-3] binding exposed port n4:8091/tcp -> 172.17.0.1:46454/tcp 2021-06-09T14:17:04.719-07:00 INFO DockerTestBase [pool-2] binding exposed port n3:8091/tcp -> 172.17.0.1:41546/tcp 2021-06-09T14:17:04.720-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8095/tcp -> 172.17.0.1:38040/tcp 2021-06-09T14:17:04.720-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8095/tcp -> 172.17.0.1:35513/tcp 2021-06-09T14:17:04.721-07:00 INFO DockerTestBase [pool-2] binding exposed port n3:8095/tcp -> 172.17.0.1:40376/tcp 2021-06-09T14:17:04.721-07:00 INFO DockerTestBase [pool-3] binding exposed port n4:8095/tcp -> 172.17.0.1:34857/tcp 2021-06-09T14:17:04.727-07:00 INFO DockerTestBase [pool-4] binding exposed port n5:8091/tcp -> 172.17.0.1:41975/tcp 2021-06-09T14:17:04.727-07:00 INFO DockerTestBase [pool-4] binding exposed port n5:8095/tcp -> 172.17.0.1:41338/tcp 2021-06-09T14:17:04.985-07:00 INFO DockerTestBase [pool-4] Starting docker container: e0fa69a0e7221 2021-06-09T14:17:04.988-07:00 INFO DockerTestBase [pool-3] Starting docker container: b308b92b68bde 2021-06-09T14:17:04.990-07:00 INFO DockerTestBase [pool-1] Starting docker container: 6acaa4e92a2ac 2021-06-09T14:17:05.007-07:00 INFO DockerTestBase [pool-2] Starting docker container: 4a4235c81dc86 2021-06-09T14:17:05.069-07:00 INFO DockerTestBase [pool-0] Starting docker container: 6a9dbbdc86f03 2021-06-09T14:17:07.034-07:00 INFO DockerTestBase [pool-3] Started docker container: b308b92b68bde 2021-06-09T14:17:07.237-07:00 INFO DockerTestBase [pool-4] Started docker container: e0fa69a0e7221 2021-06-09T14:17:07.370-07:00 INFO DockerTestBase [pool-1] Started docker container: 6acaa4e92a2ac 2021-06-09T14:17:07.447-07:00 INFO DockerTestBase [pool-4] Container information for e0fa69a0e7221: host: n5.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:41338->192.168.176.4:8095, 172.17.0.1:41975->192.168.176.4:8091 2021-06-09T14:17:07.448-07:00 INFO DockerTestBase [pool-1] Container information for 6acaa4e92a2ac: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:38040->192.168.176.3:8095, 172.17.0.1:38319->192.168.176.3:8091 2021-06-09T14:17:07.452-07:00 INFO DockerTestBase [pool-2] Started docker container: 4a4235c81dc86 2021-06-09T14:17:07.533-07:00 INFO DockerTestBase [pool-2] Container information for 4a4235c81dc86: host: n3.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.5, 172.17.0.1:40376->192.168.176.5:8095, 172.17.0.1:41546->192.168.176.5:8091 2021-06-09T14:17:07.541-07:00 INFO DockerTestBase [pool-0] Started docker container: 6a9dbbdc86f03 2021-06-09T14:17:07.549-07:00 INFO DockerTestBase [pool-3] Container information for b308b92b68bde: host: n4.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:34857->192.168.176.2:8095, 172.17.0.1:46454->192.168.176.2:8091 2021-06-09T14:17:07.595-07:00 INFO DockerTestBase [pool-0] Container information for 6a9dbbdc86f03: host: n1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.6, 172.17.0.1:35513->192.168.176.6:8095, 172.17.0.1:45949->192.168.176.6:8091 2021-06-09T14:17:22.909-07:00 INFO DockerTestBase [pool-3] Running 'node-init' on node n4 (192.168.176.2) 2021-06-09T14:17:22.918-07:00 INFO DockerTestBase [pool-3] Executing the command: docker exec n4.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n4.couchbase.host 2021-06-09T14:17:23.973-07:00 INFO DockerTestBase [pool-4] Running 'node-init' on node n5 (192.168.176.4) 2021-06-09T14:17:23.973-07:00 INFO DockerTestBase [pool-4] Executing the command: docker exec n5.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n5.couchbase.host 2021-06-09T14:17:24.283-07:00 INFO DockerTestBase [pool-3+] >> SUCCESS: Node initialized 2021-06-09T14:17:24.719-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node n3 (192.168.176.5) 2021-06-09T14:17:24.720-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname n3.couchbase.host 2021-06-09T14:17:25.388-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1 (192.168.176.6) 2021-06-09T14:17:25.389-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.6:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:17:26.155-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T14:17:26.363-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:17:26.427-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node n1 (192.168.176.6) with services: [DATA] 2021-06-09T14:17:26.429-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:17:26.446-07:00 INFO DockerTestBase [pool-4+] >> SUCCESS: Node initialized 2021-06-09T14:17:27.049-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:17:27.108-07:00 INFO DockerTestBase [main] Taking cbcollect_info from n2 (192.168.176.3) 2021-06-09T14:17:27.108-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Taking cbcollect_info from n5 (192.168.176.4) 2021-06-09T14:17:27.109-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Executing the command: docker exec n5.couchbase.host sh -c ls /opt/couchbase/var/lib/couchbase/logs/cbcollect_info*.zip 2>/dev/null | wc -l 2021-06-09T14:17:27.109-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host sh -c ls /opt/couchbase/var/lib/couchbase/logs/cbcollect_info*.zip 2>/dev/null | wc -l 2021-06-09T14:17:27.110-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Taking cbcollect_info from n4 (192.168.176.2) 2021-06-09T14:17:27.110-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Executing the command: docker exec n4.couchbase.host sh -c ls /opt/couchbase/var/lib/couchbase/logs/cbcollect_info*.zip 2>/dev/null | wc -l 2021-06-09T14:17:27.111-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Taking cbcollect_info from n1 (192.168.176.6) 2021-06-09T14:17:27.115-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Executing the command: docker exec n1.couchbase.host sh -c ls /opt/couchbase/var/lib/couchbase/logs/cbcollect_info*.zip 2>/dev/null | wc -l 2021-06-09T14:17:27.365-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5+] >> 0 2021-06-09T14:17:27.378-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Executing the command: docker exec n4.couchbase.host /opt/couchbase/bin/cbcollect_info /opt/couchbase/var/lib/couchbase/logs/cbcollect_info.zip 2021-06-09T14:17:27.390-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> 0 2021-06-09T14:17:27.396-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Executing the command: docker exec n5.couchbase.host /opt/couchbase/bin/cbcollect_info /opt/couchbase/var/lib/couchbase/logs/cbcollect_info.zip 2021-06-09T14:17:27.423-07:00 INFO DockerTestBase [main+] >> 0 2021-06-09T14:17:27.432-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/cbcollect_info /opt/couchbase/var/lib/couchbase/logs/cbcollect_info.zip 2021-06-09T14:17:27.470-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7+] >> 0 2021-06-09T14:17:27.482-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/cbcollect_info /opt/couchbase/var/lib/couchbase/logs/cbcollect_info.zip 2021-06-09T14:19:37.146-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Finished taking cbcollect_info from n4 (192.168.176.2) 2021-06-09T14:19:37.147-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container b308b92b68bde: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0003_n4_thisbuild 2021-06-09T14:19:37.294-07:00 INFO DockerTestBase [main] Finished taking cbcollect_info from n2 (192.168.176.3) 2021-06-09T14:19:37.295-07:00 INFO DockerTestBase [main] Copying files from docker container 6acaa4e92a2ac: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0002_n2_thisbuild 2021-06-09T14:19:37.422-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Finished taking cbcollect_info from n5 (192.168.176.4) 2021-06-09T14:19:37.423-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container e0fa69a0e7221: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0005_n5_thisbuild 2021-06-09T14:19:37.509-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: b308b92b68bde 2021-06-09T14:19:37.598-07:00 INFO DockerTestBase [main] Stopping docker container: 6acaa4e92a2ac 2021-06-09T14:19:37.715-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: e0fa69a0e7221 2021-06-09T14:19:38.134-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: b308b92b68bde 2021-06-09T14:19:38.135-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Taking cbcollect_info from n3 (192.168.176.5) 2021-06-09T14:19:38.135-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Executing the command: docker exec n3.couchbase.host sh -c ls /opt/couchbase/var/lib/couchbase/logs/cbcollect_info*.zip 2>/dev/null | wc -l 2021-06-09T14:19:38.297-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5+] >> 0 2021-06-09T14:19:38.303-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/cbcollect_info /opt/couchbase/var/lib/couchbase/logs/cbcollect_info.zip 2021-06-09T14:19:38.375-07:00 INFO DockerTestBase [main] Stopped docker container: 6acaa4e92a2ac 2021-06-09T14:19:38.428-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: e0fa69a0e7221 2021-06-09T14:20:04.966-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Finished taking cbcollect_info from n1 (192.168.176.6) 2021-06-09T14:20:04.966-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Copying files from docker container 6a9dbbdc86f03: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0001_n1_thisbuild 2021-06-09T14:20:05.191-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopping docker container: 6a9dbbdc86f03 2021-06-09T14:20:05.591-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopped docker container: 6a9dbbdc86f03 2021-06-09T14:21:41.963-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Finished taking cbcollect_info from n3 (192.168.176.5) 2021-06-09T14:21:41.964-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container 4a4235c81dc86: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD/testQuorumFailover/0004_n3_thisbuild 2021-06-09T14:21:42.192-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: 4a4235c81dc86 2021-06-09T14:21:42.599-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: 4a4235c81dc86 2021-06-09T14:21:42.601-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: b308b92b68bde 2021-06-09T14:21:42.601-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: e0fa69a0e7221 2021-06-09T14:21:42.601-07:00 INFO DockerTestBase [main] Removing docker container: 6acaa4e92a2ac 2021-06-09T14:21:42.601-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 6a9dbbdc86f03 2021-06-09T14:21:42.615-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 6a9dbbdc86f03 2021-06-09T14:21:42.616-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: e0fa69a0e7221 2021-06-09T14:21:42.616-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 4a4235c81dc86 2021-06-09T14:21:42.618-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: b308b92b68bde 2021-06-09T14:21:42.626-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 4a4235c81dc86 2021-06-09T14:21:42.629-07:00 INFO DockerTestBase [main] Removed docker container: 6acaa4e92a2ac 2021-06-09T14:21:42.630-07:00 INFO DockerTestBase [main] Removing docker network: 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 2021-06-09T14:21:42.685-07:00 INFO DockerTestBase [main] Removed docker network: 3a8fc9c9c6c19b203571b398431b5776ce5d75f49b373d786189b30d7d7bbb02 2021-06-09T14:21:42.693-07:00 WARN TestMethodTracer [main] ### testQuorumFailover FAILED java.util.ConcurrentModificationException: null at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1661) ~[?:?] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?] at com.couchbase.analytics.test.docker.DockerTestBase.getBasicAuthHttpCtx(DockerTestBase.java:239) ~[test-classes/:?] at java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305) ~[?:?] at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195) ~[?:?] at java.lang.ThreadLocal.get(ThreadLocal.java:172) ~[?:?] at com.couchbase.analytics.test.common.HttpContextHelper.getHttpContext(HttpContextHelper.java:22) ~[cbas-server-7.0.0-SNAPSHOT-tests.jar:] at com.couchbase.analytics.test.common.ClusterTestBase.waitForStatus(ClusterTestBase.java:858) ~[cbas-server-7.0.0-SNAPSHOT-tests.jar:] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:577) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:453) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:428) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:421) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:415) ~[test-classes/:?] at com.couchbase.analytics.test.docker.upgrade.UpgradeTestBase.createNode(UpgradeTestBase.java:170) ~[test-classes/:?] at com.couchbase.analytics.test.docker.upgrade.UpgradeTestBase.createNode(UpgradeTestBase.java:164) ~[test-classes/:?] at com.couchbase.analytics.test.docker.upgrade.UpgradeTestBase.createNode(UpgradeTestBase.java:160) ~[test-classes/:?] at com.couchbase.analytics.test.common.ClusterTestBase.lambda$invokeAll$38(ClusterTestBase.java:1253) ~[cbas-server-7.0.0-SNAPSHOT-tests.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:834) ~[?:?] [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 281.141 s <<< FAILURE! - in com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD [ERROR] testQuorumFailover(com.couchbase.analytics.test.docker.lifecycle.QuorumFailoverITD) Time elapsed: 279.507 s <<< ERROR! java.util.ConcurrentModificationException [INFO] Running com.couchbase.analytics.test.docker.remote.CbRemoteLinks603ITD 2021-06-09 14:21:44,902 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:21:44.989-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:21:45.029-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:21:45.030-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:21:45.078-07:00 WARN TestMethodTracer [main] ### testExternal603Target START 2021-06-09T14:21:45.120-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:21:46.330-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-82dbc468-17cf-4d34-ba46-f9d56bf6d0f7) 2021-06-09T14:21:46.403-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-82dbc468-17cf-4d34-ba46-f9d56bf6d0f7, id = e45c21b7f2e76d1b8f280bd64d6a1c3238e30458f514283e338334c4055743e9 } 2021-06-09T14:21:46.414-07:00 INFO UpgradeTestBase [pool-0] Creating target node ([couchbase:enterprise-6.0.3])... 2021-06-09T14:21:46.409-07:00 INFO UpgradeTestBase [pool-1] Creating source node using 'this build' docker image... 2021-06-09T14:21:46.451-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001target to /opt/couchbase/var 2021-06-09T14:21:46.451-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002source to /opt/couchbase/var 2021-06-09T14:21:46.455-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull couchbase:enterprise-6.0.3 2021-06-09T14:21:46.457-07:00 INFO DockerTestBase [pool-1] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:21:46.585-07:00 INFO DockerTestBase [pool-1+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:21:46.586-07:00 INFO DockerTestBase [pool-1+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:21:46.587-07:00 INFO DockerTestBase [pool-1+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:21:46.589-07:00 INFO DockerTestBase [pool-1+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:21:46.593-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks603ITD/testExternal603Target/0002_source_thisbuild, network = { name = cbas-docker-test-0001-82dbc468-17cf-4d34-ba46-f9d56bf6d0f7, id = e45c21b7f2e76d1b8f280bd64d6a1c3238e30458f514283e338334c4055743e9 }) 2021-06-09T14:21:46.630-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8091/tcp -> 172.17.0.1:37527/tcp 2021-06-09T14:21:46.631-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8095/tcp -> 172.17.0.1:40795/tcp 2021-06-09T14:21:46.755-07:00 INFO DockerTestBase [pool-1] Starting docker container: ff2592a3e3f4a 2021-06-09T14:21:47.492-07:00 INFO DockerTestBase [pool-1] Started docker container: ff2592a3e3f4a 2021-06-09T14:21:47.633-07:00 INFO DockerTestBase [pool-1] Container information for ff2592a3e3f4a: host: source.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:40795->192.168.176.2:8095, 172.17.0.1:37527->192.168.176.2:8091 2021-06-09T14:21:47.843-07:00 INFO DockerTestBase [pool-0+] >> enterprise-6.0.3: Pulling from library/couchbase 2021-06-09T14:21:48.207-07:00 INFO DockerTestBase [pool-0+] >> 4bbfd2c87b75: Already exists 2021-06-09T14:21:48.209-07:00 INFO DockerTestBase [pool-0+] >> d2e110be24e1: Already exists 2021-06-09T14:21:48.210-07:00 INFO DockerTestBase [pool-0+] >> 889a7173dcfe: Already exists 2021-06-09T14:21:48.213-07:00 INFO DockerTestBase [pool-0+] >> cf4443906eb6: Pulling fs layer 2021-06-09T14:21:48.214-07:00 INFO DockerTestBase [pool-0+] >> 1810bc668c2f: Pulling fs layer 2021-06-09T14:21:48.214-07:00 INFO DockerTestBase [pool-0+] >> eb56578c4492: Pulling fs layer 2021-06-09T14:21:48.214-07:00 INFO DockerTestBase [pool-0+] >> bcda223ee43e: Pulling fs layer 2021-06-09T14:21:48.214-07:00 INFO DockerTestBase [pool-0+] >> bf1a544f8b41: Pulling fs layer 2021-06-09T14:21:48.215-07:00 INFO DockerTestBase [pool-0+] >> 9bf71193f9e3: Pulling fs layer 2021-06-09T14:21:48.215-07:00 INFO DockerTestBase [pool-0+] >> ec94c94027a5: Pulling fs layer 2021-06-09T14:21:48.215-07:00 INFO DockerTestBase [pool-0+] >> 5318c19264db: Pulling fs layer 2021-06-09T14:21:48.215-07:00 INFO DockerTestBase [pool-0+] >> 8abc247dffd1: Pulling fs layer 2021-06-09T14:21:48.216-07:00 INFO DockerTestBase [pool-0+] >> 04fa7b28a976: Pulling fs layer 2021-06-09T14:21:48.216-07:00 INFO DockerTestBase [pool-0+] >> b6f386b284da: Pulling fs layer 2021-06-09T14:21:48.216-07:00 INFO DockerTestBase [pool-0+] >> 9bf71193f9e3: Waiting 2021-06-09T14:21:48.216-07:00 INFO DockerTestBase [pool-0+] >> 8abc247dffd1: Waiting 2021-06-09T14:21:48.216-07:00 INFO DockerTestBase [pool-0+] >> 04fa7b28a976: Waiting 2021-06-09T14:21:48.216-07:00 INFO DockerTestBase [pool-0+] >> ec94c94027a5: Waiting 2021-06-09T14:21:48.217-07:00 INFO DockerTestBase [pool-0+] >> 5318c19264db: Waiting 2021-06-09T14:21:48.217-07:00 INFO DockerTestBase [pool-0+] >> b6f386b284da: Waiting 2021-06-09T14:21:48.217-07:00 INFO DockerTestBase [pool-0+] >> bcda223ee43e: Waiting 2021-06-09T14:21:48.217-07:00 INFO DockerTestBase [pool-0+] >> bf1a544f8b41: Waiting 2021-06-09T14:21:48.572-07:00 INFO DockerTestBase [pool-0+] >> 1810bc668c2f: Verifying Checksum 2021-06-09T14:21:48.572-07:00 INFO DockerTestBase [pool-0+] >> 1810bc668c2f: Download complete 2021-06-09T14:21:48.575-07:00 INFO DockerTestBase [pool-0+] >> eb56578c4492: Download complete 2021-06-09T14:21:48.773-07:00 INFO DockerTestBase [pool-0+] >> cf4443906eb6: Verifying Checksum 2021-06-09T14:21:48.773-07:00 INFO DockerTestBase [pool-0+] >> cf4443906eb6: Download complete 2021-06-09T14:21:48.939-07:00 INFO DockerTestBase [pool-0+] >> bf1a544f8b41: Verifying Checksum 2021-06-09T14:21:48.940-07:00 INFO DockerTestBase [pool-0+] >> bf1a544f8b41: Download complete 2021-06-09T14:21:49.133-07:00 INFO DockerTestBase [pool-0+] >> 9bf71193f9e3: Verifying Checksum 2021-06-09T14:21:49.134-07:00 INFO DockerTestBase [pool-0+] >> 9bf71193f9e3: Download complete 2021-06-09T14:21:49.270-07:00 INFO DockerTestBase [pool-0+] >> ec94c94027a5: Verifying Checksum 2021-06-09T14:21:49.270-07:00 INFO DockerTestBase [pool-0+] >> ec94c94027a5: Download complete 2021-06-09T14:21:49.481-07:00 INFO DockerTestBase [pool-0+] >> 5318c19264db: Verifying Checksum 2021-06-09T14:21:49.481-07:00 INFO DockerTestBase [pool-0+] >> 5318c19264db: Download complete 2021-06-09T14:21:49.622-07:00 INFO DockerTestBase [pool-0+] >> 8abc247dffd1: Verifying Checksum 2021-06-09T14:21:49.629-07:00 INFO DockerTestBase [pool-0+] >> 8abc247dffd1: Download complete 2021-06-09T14:21:49.828-07:00 INFO DockerTestBase [pool-0+] >> 04fa7b28a976: Verifying Checksum 2021-06-09T14:21:49.828-07:00 INFO DockerTestBase [pool-0+] >> 04fa7b28a976: Download complete 2021-06-09T14:21:49.956-07:00 INFO DockerTestBase [pool-0+] >> b6f386b284da: Verifying Checksum 2021-06-09T14:21:49.956-07:00 INFO DockerTestBase [pool-0+] >> b6f386b284da: Download complete 2021-06-09T14:21:50.762-07:00 INFO DockerTestBase [pool-0+] >> cf4443906eb6: Pull complete 2021-06-09T14:21:50.888-07:00 INFO DockerTestBase [pool-0+] >> 1810bc668c2f: Pull complete 2021-06-09T14:21:51.025-07:00 INFO DockerTestBase [pool-0+] >> eb56578c4492: Pull complete 2021-06-09T14:21:54.562-07:00 INFO DockerTestBase [pool-0+] >> bcda223ee43e: Verifying Checksum 2021-06-09T14:21:54.562-07:00 INFO DockerTestBase [pool-0+] >> bcda223ee43e: Download complete 2021-06-09T14:21:57.996-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node source (192.168.176.2) 2021-06-09T14:21:57.998-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname source.couchbase.host 2021-06-09T14:21:59.212-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:21:59.243-07:00 INFO DockerTestBase [pool-1] Running 'cluster-init' on node source (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T14:21:59.244-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:21:59.719-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Cluster initialized 2021-06-09T14:21:59.749-07:00 INFO DockerTestBase [pool-1] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:22:05.305-07:00 INFO DockerTestBase [pool-0+] >> bcda223ee43e: Pull complete 2021-06-09T14:22:05.436-07:00 INFO DockerTestBase [pool-0+] >> bf1a544f8b41: Pull complete 2021-06-09T14:22:05.568-07:00 INFO DockerTestBase [pool-0+] >> 9bf71193f9e3: Pull complete 2021-06-09T14:22:05.704-07:00 INFO DockerTestBase [pool-0+] >> ec94c94027a5: Pull complete 2021-06-09T14:22:05.832-07:00 INFO DockerTestBase [pool-0+] >> 5318c19264db: Pull complete 2021-06-09T14:22:05.968-07:00 INFO DockerTestBase [pool-0+] >> 8abc247dffd1: Pull complete 2021-06-09T14:22:06.096-07:00 INFO DockerTestBase [pool-0+] >> 04fa7b28a976: Pull complete 2021-06-09T14:22:06.224-07:00 INFO DockerTestBase [pool-0+] >> b6f386b284da: Pull complete 2021-06-09T14:22:06.232-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:61e4543f38022a6e76683a5002536243e2af53e75fa23a72613f828100657b4d 2021-06-09T14:22:06.233-07:00 INFO DockerTestBase [pool-0+] >> Status: Downloaded newer image for couchbase:enterprise-6.0.3 2021-06-09T14:22:06.237-07:00 INFO DockerTestBase [pool-0+] >> docker.io/library/couchbase:enterprise-6.0.3 2021-06-09T14:22:06.241-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks603ITD/testExternal603Target/0001_target_couchbase_enterprise-6.0.3, network = { name = cbas-docker-test-0001-82dbc468-17cf-4d34-ba46-f9d56bf6d0f7, id = e45c21b7f2e76d1b8f280bd64d6a1c3238e30458f514283e338334c4055743e9 }) 2021-06-09T14:22:06.242-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8091/tcp -> 172.17.0.1:43076/tcp 2021-06-09T14:22:06.242-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8095/tcp -> 172.17.0.1:36372/tcp 2021-06-09T14:22:06.328-07:00 INFO DockerTestBase [pool-0] Starting docker container: f639897955053 2021-06-09T14:22:07.219-07:00 INFO DockerTestBase [pool-0] Started docker container: f639897955053 2021-06-09T14:22:07.237-07:00 INFO DockerTestBase [pool-0] Container information for f639897955053: host: target.couchbase.host, image: enterprise-6.0.3, ip: 192.168.176.3, 172.17.0.1:36372->192.168.176.3:8095, 172.17.0.1:43076->192.168.176.3:8091 2021-06-09T14:22:17.286-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node target (192.168.176.3) 2021-06-09T14:22:17.286-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname target.couchbase.host 2021-06-09T14:22:17.901-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:22:17.912-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node target (192.168.176.3) with services: [DATA] 2021-06-09T14:22:17.912-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:22:18.271-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:22:18.322-07:00 INFO KvStoreHttpHelper [pool-0] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:43076] 2021-06-09T14:22:18.325-07:00 INFO KvStoreHttpHelper [pool-0] installing sample(s): [beer-sample] on 172.17.0.1:43076 2021-06-09T14:22:18.411-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37527] 2021-06-09T14:22:18.414-07:00 INFO DockerTestBase [main] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.2 -u couchbase -p couchbase --create --dataverse Default --name externalcb --type couchbase --hostname target.couchbase.host --link-password couchbase --encryption none --link-username couchbase 2021-06-09T14:22:18.979-07:00 INFO DockerTestBase [main+] >> DEPRECATED: --dataverse is deprecated, please use --scope instead 2021-06-09T14:22:18.979-07:00 INFO DockerTestBase [main+] >> SUCCESS: Link created 2021-06-09T14:22:22.452-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:22:22.455-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:22:23.079-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:22:23.084-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:22:24.085-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:22:24.215-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3433 } 2021-06-09T14:22:24.216-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 3433 } 2021-06-09T14:22:25.217-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:22:25.329-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:22:25.329-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:22:25.335-07:00 INFO DockerTestBase [main] Copying files from docker container f639897955053: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks603ITD/testExternal603Target/0001_target_couchbase_enterprise-6.0.3 2021-06-09T14:22:25.337-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container ff2592a3e3f4a: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks603ITD/testExternal603Target/0002_source_thisbuild 2021-06-09T14:22:25.608-07:00 INFO DockerTestBase [main] Stopping docker container: f639897955053 2021-06-09T14:22:25.691-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: ff2592a3e3f4a 2021-06-09T14:22:26.102-07:00 INFO DockerTestBase [main] Stopped docker container: f639897955053 2021-06-09T14:22:26.142-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: ff2592a3e3f4a 2021-06-09T14:22:26.144-07:00 INFO DockerTestBase [main] Removing docker container: f639897955053 2021-06-09T14:22:26.144-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: ff2592a3e3f4a 2021-06-09T14:22:26.169-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: ff2592a3e3f4a 2021-06-09T14:22:26.172-07:00 INFO DockerTestBase [main] Removed docker container: f639897955053 2021-06-09T14:22:26.174-07:00 INFO DockerTestBase [main] Removing docker network: e45c21b7f2e76d1b8f280bd64d6a1c3238e30458f514283e338334c4055743e9 2021-06-09T14:22:26.266-07:00 INFO DockerTestBase [main] Removed docker network: e45c21b7f2e76d1b8f280bd64d6a1c3238e30458f514283e338334c4055743e9 2021-06-09T14:22:26.272-07:00 WARN TestMethodTracer [main] ### testExternal603Target SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.688 s - in com.couchbase.analytics.test.docker.remote.CbRemoteLinks603ITD [INFO] Running com.couchbase.analytics.test.docker.remote.CbRemoteLinks651ITD 2021-06-09 14:22:28,377 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:22:28.461-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:22:28.497-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:22:28.497-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:22:28.586-07:00 WARN TestMethodTracer [main] ### testRemoteCluster START 2021-06-09T14:22:28.623-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:22:29.776-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-d7248c62-c768-4644-af7d-bc21f90985a2) 2021-06-09T14:22:29.848-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-d7248c62-c768-4644-af7d-bc21f90985a2, id = 5b4968997f981aa33976bf8f95617731de538afa6c2e028da21d699c208bfa68 } 2021-06-09T14:22:29.853-07:00 INFO UpgradeTestBase [pool-1] Creating source node using 'this build' docker image... 2021-06-09T14:22:29.864-07:00 INFO UpgradeTestBase [pool-0] Creating target node ([couchbase:enterprise-6.5.1])... 2021-06-09T14:22:29.870-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002target to /opt/couchbase/var 2021-06-09T14:22:29.870-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001source to /opt/couchbase/var 2021-06-09T14:22:29.877-07:00 INFO DockerTestBase [pool-1] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:22:29.879-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull couchbase:enterprise-6.5.1 2021-06-09T14:22:30.121-07:00 INFO DockerTestBase [pool-1+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:22:30.170-07:00 INFO DockerTestBase [pool-1+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:22:30.171-07:00 INFO DockerTestBase [pool-1+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:22:30.221-07:00 INFO DockerTestBase [pool-1+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:22:30.224-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks651ITD/testRemoteCluster/0001_source_thisbuild, network = { name = cbas-docker-test-0001-d7248c62-c768-4644-af7d-bc21f90985a2, id = 5b4968997f981aa33976bf8f95617731de538afa6c2e028da21d699c208bfa68 }) 2021-06-09T14:22:30.248-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8091/tcp -> 172.17.0.1:37257/tcp 2021-06-09T14:22:30.248-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8095/tcp -> 172.17.0.1:33344/tcp 2021-06-09T14:22:30.921-07:00 INFO DockerTestBase [pool-1] Starting docker container: af0e00d42ab8b 2021-06-09T14:22:31.997-07:00 INFO DockerTestBase [pool-0+] >> enterprise-6.5.1: Pulling from library/couchbase 2021-06-09T14:22:33.025-07:00 INFO DockerTestBase [pool-0+] >> 4bbfd2c87b75: Already exists 2021-06-09T14:22:33.028-07:00 INFO DockerTestBase [pool-0+] >> d2e110be24e1: Already exists 2021-06-09T14:22:33.029-07:00 INFO DockerTestBase [pool-0+] >> 889a7173dcfe: Already exists 2021-06-09T14:22:33.032-07:00 INFO DockerTestBase [pool-0+] >> f8723b0ce09e: Already exists 2021-06-09T14:22:33.034-07:00 INFO DockerTestBase [pool-0+] >> 5ad9d59e6d21: Already exists 2021-06-09T14:22:33.038-07:00 INFO DockerTestBase [pool-0+] >> c0e7c3620841: Pulling fs layer 2021-06-09T14:22:33.038-07:00 INFO DockerTestBase [pool-0+] >> 2fdc5964e843: Pulling fs layer 2021-06-09T14:22:33.038-07:00 INFO DockerTestBase [pool-0+] >> 64dc033078d3: Pulling fs layer 2021-06-09T14:22:33.039-07:00 INFO DockerTestBase [pool-0+] >> d6e62a708e1b: Pulling fs layer 2021-06-09T14:22:33.039-07:00 INFO DockerTestBase [pool-0+] >> d83bb838adf8: Pulling fs layer 2021-06-09T14:22:33.039-07:00 INFO DockerTestBase [pool-0+] >> 2df885e4496a: Pulling fs layer 2021-06-09T14:22:33.039-07:00 INFO DockerTestBase [pool-0+] >> 454dacb0c733: Pulling fs layer 2021-06-09T14:22:33.039-07:00 INFO DockerTestBase [pool-0+] >> 7f3cfad73a73: Pulling fs layer 2021-06-09T14:22:33.040-07:00 INFO DockerTestBase [pool-0+] >> c327bff855f6: Pulling fs layer 2021-06-09T14:22:33.040-07:00 INFO DockerTestBase [pool-0+] >> d6e62a708e1b: Waiting 2021-06-09T14:22:33.040-07:00 INFO DockerTestBase [pool-0+] >> d83bb838adf8: Waiting 2021-06-09T14:22:33.040-07:00 INFO DockerTestBase [pool-0+] >> 2df885e4496a: Waiting 2021-06-09T14:22:33.040-07:00 INFO DockerTestBase [pool-0+] >> 454dacb0c733: Waiting 2021-06-09T14:22:33.040-07:00 INFO DockerTestBase [pool-0+] >> 7f3cfad73a73: Waiting 2021-06-09T14:22:33.041-07:00 INFO DockerTestBase [pool-0+] >> c327bff855f6: Waiting 2021-06-09T14:22:33.384-07:00 INFO DockerTestBase [pool-0+] >> c0e7c3620841: Verifying Checksum 2021-06-09T14:22:33.384-07:00 INFO DockerTestBase [pool-0+] >> c0e7c3620841: Download complete 2021-06-09T14:22:33.400-07:00 INFO DockerTestBase [pool-1] Started docker container: af0e00d42ab8b 2021-06-09T14:22:33.402-07:00 INFO DockerTestBase [pool-0+] >> 64dc033078d3: Verifying Checksum 2021-06-09T14:22:33.403-07:00 INFO DockerTestBase [pool-0+] >> 64dc033078d3: Download complete 2021-06-09T14:22:33.670-07:00 INFO DockerTestBase [pool-1] Container information for af0e00d42ab8b: host: source.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:37257->192.168.176.2:8091, 172.17.0.1:33344->192.168.176.2:8095 2021-06-09T14:22:33.750-07:00 INFO DockerTestBase [pool-0+] >> d6e62a708e1b: Verifying Checksum 2021-06-09T14:22:33.750-07:00 INFO DockerTestBase [pool-0+] >> d6e62a708e1b: Download complete 2021-06-09T14:22:33.771-07:00 INFO DockerTestBase [pool-0+] >> c0e7c3620841: Pull complete 2021-06-09T14:22:33.813-07:00 INFO DockerTestBase [pool-0+] >> d83bb838adf8: Verifying Checksum 2021-06-09T14:22:33.813-07:00 INFO DockerTestBase [pool-0+] >> d83bb838adf8: Download complete 2021-06-09T14:22:34.102-07:00 INFO DockerTestBase [pool-0+] >> 2df885e4496a: Verifying Checksum 2021-06-09T14:22:34.103-07:00 INFO DockerTestBase [pool-0+] >> 2df885e4496a: Download complete 2021-06-09T14:22:34.192-07:00 INFO DockerTestBase [pool-0+] >> 454dacb0c733: Download complete 2021-06-09T14:22:34.469-07:00 INFO DockerTestBase [pool-0+] >> 7f3cfad73a73: Verifying Checksum 2021-06-09T14:22:34.470-07:00 INFO DockerTestBase [pool-0+] >> 7f3cfad73a73: Download complete 2021-06-09T14:22:34.563-07:00 INFO DockerTestBase [pool-0+] >> c327bff855f6: Verifying Checksum 2021-06-09T14:22:34.563-07:00 INFO DockerTestBase [pool-0+] >> c327bff855f6: Download complete 2021-06-09T14:22:39.085-07:00 INFO DockerTestBase [pool-0+] >> 2fdc5964e843: Verifying Checksum 2021-06-09T14:22:39.085-07:00 INFO DockerTestBase [pool-0+] >> 2fdc5964e843: Download complete 2021-06-09T14:22:47.154-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node source (192.168.176.2) 2021-06-09T14:22:47.157-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname source.couchbase.host 2021-06-09T14:22:48.259-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:22:48.290-07:00 INFO DockerTestBase [pool-1] Running 'cluster-init' on node source (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T14:22:48.290-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:22:48.800-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Cluster initialized 2021-06-09T14:22:48.841-07:00 INFO DockerTestBase [pool-1] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:22:52.960-07:00 INFO DockerTestBase [pool-0+] >> 2fdc5964e843: Pull complete 2021-06-09T14:22:53.060-07:00 INFO DockerTestBase [pool-0+] >> 64dc033078d3: Pull complete 2021-06-09T14:22:53.184-07:00 INFO DockerTestBase [pool-0+] >> d6e62a708e1b: Pull complete 2021-06-09T14:22:53.308-07:00 INFO DockerTestBase [pool-0+] >> d83bb838adf8: Pull complete 2021-06-09T14:22:53.412-07:00 INFO DockerTestBase [pool-0+] >> 2df885e4496a: Pull complete 2021-06-09T14:22:53.527-07:00 INFO DockerTestBase [pool-0+] >> 454dacb0c733: Pull complete 2021-06-09T14:22:53.628-07:00 INFO DockerTestBase [pool-0+] >> 7f3cfad73a73: Pull complete 2021-06-09T14:22:53.731-07:00 INFO DockerTestBase [pool-0+] >> c327bff855f6: Pull complete 2021-06-09T14:22:53.738-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:e5a07ee624c264d3a8ebcc2ff0989ea710c5843f73b0d541b2e3ea2f476adc13 2021-06-09T14:22:53.740-07:00 INFO DockerTestBase [pool-0+] >> Status: Downloaded newer image for couchbase:enterprise-6.5.1 2021-06-09T14:22:53.742-07:00 INFO DockerTestBase [pool-0+] >> docker.io/library/couchbase:enterprise-6.5.1 2021-06-09T14:22:53.744-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks651ITD/testRemoteCluster/0002_target_couchbase_enterprise-6.5.1, network = { name = cbas-docker-test-0001-d7248c62-c768-4644-af7d-bc21f90985a2, id = 5b4968997f981aa33976bf8f95617731de538afa6c2e028da21d699c208bfa68 }) 2021-06-09T14:22:53.745-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8091/tcp -> 172.17.0.1:35105/tcp 2021-06-09T14:22:53.745-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8095/tcp -> 172.17.0.1:38298/tcp 2021-06-09T14:22:53.803-07:00 INFO DockerTestBase [pool-0] Starting docker container: 5d07559d4f4d3 2021-06-09T14:22:54.534-07:00 INFO DockerTestBase [pool-0] Started docker container: 5d07559d4f4d3 2021-06-09T14:22:54.542-07:00 INFO DockerTestBase [pool-0] Container information for 5d07559d4f4d3: host: target.couchbase.host, image: enterprise-6.5.1, ip: 192.168.176.3, 172.17.0.1:35105->192.168.176.3:8091, 172.17.0.1:38298->192.168.176.3:8095 2021-06-09T14:23:07.241-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node target (192.168.176.3) 2021-06-09T14:23:07.242-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname target.couchbase.host 2021-06-09T14:23:10.436-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:23:10.460-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node target (192.168.176.3) with services: [DATA] 2021-06-09T14:23:10.460-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:23:10.892-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:23:10.960-07:00 INFO KvStoreHttpHelper [pool-0] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35105] 2021-06-09T14:23:10.964-07:00 INFO KvStoreHttpHelper [pool-0] installing sample(s): [beer-sample] on 172.17.0.1:35105 2021-06-09T14:23:12.587-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37257] 2021-06-09T14:23:12.590-07:00 INFO DockerTestBase [main] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.2 -u couchbase -p couchbase --create --dataverse Default --name externalcb --type couchbase --hostname target.couchbase.host --link-password couchbase --encryption half --link-username couchbase 2021-06-09T14:23:13.309-07:00 INFO DockerTestBase [main+] >> DEPRECATED: --dataverse is deprecated, please use --scope instead 2021-06-09T14:23:13.310-07:00 INFO DockerTestBase [main+] >> SUCCESS: Link created 2021-06-09T14:23:16.621-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:23:16.624-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:23:17.115-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:23:17.121-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:23:18.123-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:23:18.291-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2487 } 2021-06-09T14:23:18.291-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2487 } 2021-06-09T14:23:19.292-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:23:19.413-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:23:19.414-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:23:19.417-07:00 INFO DockerTestBase [main] Copying files from docker container 5d07559d4f4d3: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks651ITD/testRemoteCluster/0002_target_couchbase_enterprise-6.5.1 2021-06-09T14:23:19.417-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container af0e00d42ab8b: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks651ITD/testRemoteCluster/0001_source_thisbuild 2021-06-09T14:23:19.828-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: af0e00d42ab8b 2021-06-09T14:23:19.841-07:00 INFO DockerTestBase [main] Stopping docker container: 5d07559d4f4d3 2021-06-09T14:23:20.584-07:00 INFO DockerTestBase [main] Stopped docker container: 5d07559d4f4d3 2021-06-09T14:23:20.624-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: af0e00d42ab8b 2021-06-09T14:23:20.626-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: af0e00d42ab8b 2021-06-09T14:23:20.626-07:00 INFO DockerTestBase [main] Removing docker container: 5d07559d4f4d3 2021-06-09T14:23:20.662-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: af0e00d42ab8b 2021-06-09T14:23:20.662-07:00 INFO DockerTestBase [main] Removed docker container: 5d07559d4f4d3 2021-06-09T14:23:20.664-07:00 INFO DockerTestBase [main] Removing docker network: 5b4968997f981aa33976bf8f95617731de538afa6c2e028da21d699c208bfa68 2021-06-09T14:23:20.808-07:00 INFO DockerTestBase [main] Removed docker network: 5b4968997f981aa33976bf8f95617731de538afa6c2e028da21d699c208bfa68 2021-06-09T14:23:20.814-07:00 WARN TestMethodTracer [main] ### testRemoteCluster SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.657 s - in com.couchbase.analytics.test.docker.remote.CbRemoteLinks651ITD [INFO] Running com.couchbase.analytics.test.docker.remote.CbRemoteLinks660CEITD 2021-06-09 14:23:22,863 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:23:22.944-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:23:22.979-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:23:22.979-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:23:23.024-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:community-6.6.0 2021-06-09T14:23:24.532-07:00 INFO DockerTestBase [main+] >> community-6.6.0: Pulling from library/couchbase 2021-06-09T14:23:24.995-07:00 INFO DockerTestBase [main+] >> 4bbfd2c87b75: Already exists 2021-06-09T14:23:25.082-07:00 INFO DockerTestBase [main+] >> d2e110be24e1: Already exists 2021-06-09T14:23:25.166-07:00 INFO DockerTestBase [main+] >> 889a7173dcfe: Already exists 2021-06-09T14:23:25.250-07:00 INFO DockerTestBase [main+] >> f8723b0ce09e: Already exists 2021-06-09T14:23:25.345-07:00 INFO DockerTestBase [main+] >> 5ad9d59e6d21: Already exists 2021-06-09T14:23:25.451-07:00 INFO DockerTestBase [main+] >> 06a0a05f60c2: Pulling fs layer 2021-06-09T14:23:25.451-07:00 INFO DockerTestBase [main+] >> 23a39eace359: Pulling fs layer 2021-06-09T14:23:25.451-07:00 INFO DockerTestBase [main+] >> 3cbb3cd74cdf: Pulling fs layer 2021-06-09T14:23:25.452-07:00 INFO DockerTestBase [main+] >> 796288503685: Pulling fs layer 2021-06-09T14:23:25.452-07:00 INFO DockerTestBase [main+] >> 77ea1a4ae847: Pulling fs layer 2021-06-09T14:23:25.452-07:00 INFO DockerTestBase [main+] >> 429ce3419215: Pulling fs layer 2021-06-09T14:23:25.453-07:00 INFO DockerTestBase [main+] >> 9a64e4d57d43: Pulling fs layer 2021-06-09T14:23:25.454-07:00 INFO DockerTestBase [main+] >> 796288503685: Waiting 2021-06-09T14:23:25.454-07:00 INFO DockerTestBase [main+] >> 429ce3419215: Waiting 2021-06-09T14:23:25.454-07:00 INFO DockerTestBase [main+] >> 77ea1a4ae847: Waiting 2021-06-09T14:23:25.455-07:00 INFO DockerTestBase [main+] >> 1545e3d95d2b: Pulling fs layer 2021-06-09T14:23:25.455-07:00 INFO DockerTestBase [main+] >> f7512fd814d9: Pulling fs layer 2021-06-09T14:23:25.455-07:00 INFO DockerTestBase [main+] >> 9a64e4d57d43: Waiting 2021-06-09T14:23:25.456-07:00 INFO DockerTestBase [main+] >> 1545e3d95d2b: Waiting 2021-06-09T14:23:25.456-07:00 INFO DockerTestBase [main+] >> f7512fd814d9: Waiting 2021-06-09T14:23:25.792-07:00 INFO DockerTestBase [main+] >> 06a0a05f60c2: Download complete 2021-06-09T14:23:25.807-07:00 INFO DockerTestBase [main+] >> 3cbb3cd74cdf: Verifying Checksum 2021-06-09T14:23:25.807-07:00 INFO DockerTestBase [main+] >> 3cbb3cd74cdf: Download complete 2021-06-09T14:23:26.142-07:00 INFO DockerTestBase [main+] >> 77ea1a4ae847: Verifying Checksum 2021-06-09T14:23:26.143-07:00 INFO DockerTestBase [main+] >> 77ea1a4ae847: Download complete 2021-06-09T14:23:26.151-07:00 INFO DockerTestBase [main+] >> 796288503685: Verifying Checksum 2021-06-09T14:23:26.152-07:00 INFO DockerTestBase [main+] >> 796288503685: Download complete 2021-06-09T14:23:26.152-07:00 INFO DockerTestBase [main+] >> 06a0a05f60c2: Pull complete 2021-06-09T14:23:26.506-07:00 INFO DockerTestBase [main+] >> 9a64e4d57d43: Verifying Checksum 2021-06-09T14:23:26.507-07:00 INFO DockerTestBase [main+] >> 9a64e4d57d43: Download complete 2021-06-09T14:23:26.508-07:00 INFO DockerTestBase [main+] >> 429ce3419215: Verifying Checksum 2021-06-09T14:23:26.509-07:00 INFO DockerTestBase [main+] >> 429ce3419215: Download complete 2021-06-09T14:23:26.892-07:00 INFO DockerTestBase [main+] >> f7512fd814d9: Verifying Checksum 2021-06-09T14:23:26.893-07:00 INFO DockerTestBase [main+] >> f7512fd814d9: Download complete 2021-06-09T14:23:26.898-07:00 INFO DockerTestBase [main+] >> 1545e3d95d2b: Verifying Checksum 2021-06-09T14:23:26.898-07:00 INFO DockerTestBase [main+] >> 1545e3d95d2b: Download complete 2021-06-09T14:23:29.332-07:00 INFO DockerTestBase [main+] >> 23a39eace359: Download complete 2021-06-09T14:23:37.606-07:00 INFO DockerTestBase [main+] >> 23a39eace359: Pull complete 2021-06-09T14:23:37.712-07:00 INFO DockerTestBase [main+] >> 3cbb3cd74cdf: Pull complete 2021-06-09T14:23:37.812-07:00 INFO DockerTestBase [main+] >> 796288503685: Pull complete 2021-06-09T14:23:37.916-07:00 INFO DockerTestBase [main+] >> 77ea1a4ae847: Pull complete 2021-06-09T14:23:38.024-07:00 INFO DockerTestBase [main+] >> 429ce3419215: Pull complete 2021-06-09T14:23:38.152-07:00 INFO DockerTestBase [main+] >> 9a64e4d57d43: Pull complete 2021-06-09T14:23:38.252-07:00 INFO DockerTestBase [main+] >> 1545e3d95d2b: Pull complete 2021-06-09T14:23:38.367-07:00 INFO DockerTestBase [main+] >> f7512fd814d9: Pull complete 2021-06-09T14:23:38.375-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a98074cacf4fdbfac21c1ec1b63c11cf28e0ba2715f604eb82076eaf3053eaa7 2021-06-09T14:23:38.378-07:00 INFO DockerTestBase [main+] >> Status: Downloaded newer image for couchbase:community-6.6.0 2021-06-09T14:23:38.379-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:community-6.6.0 2021-06-09T14:23:38.387-07:00 WARN TestMethodTracer [main] ### testRemoteLink START 2021-06-09T14:23:38.422-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:23:39.556-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-535c84c3-47e8-4a8f-a96e-cb441ff7606f) 2021-06-09T14:23:39.628-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-535c84c3-47e8-4a8f-a96e-cb441ff7606f, id = a5d0aa3c6d635199261eb2920150e046ffa6c25c047a2b84694e504a6f24236f } 2021-06-09T14:23:39.632-07:00 INFO UpgradeTestBase [pool-0] Creating target node ([couchbase:community-6.6.0])... 2021-06-09T14:23:39.632-07:00 INFO UpgradeTestBase [pool-1] Creating source node using 'this build' docker image... 2021-06-09T14:23:39.643-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002source to /opt/couchbase/var 2021-06-09T14:23:39.643-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001target to /opt/couchbase/var 2021-06-09T14:23:39.648-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660CEITD/testRemoteLink/0001_target_couchbase_community-6.6.0, network = { name = cbas-docker-test-0001-535c84c3-47e8-4a8f-a96e-cb441ff7606f, id = a5d0aa3c6d635199261eb2920150e046ffa6c25c047a2b84694e504a6f24236f }) 2021-06-09T14:23:39.648-07:00 INFO DockerTestBase [pool-1] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:23:39.669-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8091/tcp -> 172.17.0.1:37926/tcp 2021-06-09T14:23:39.670-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8095/tcp -> 172.17.0.1:43609/tcp 2021-06-09T14:23:39.734-07:00 INFO DockerTestBase [pool-1+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:23:39.735-07:00 INFO DockerTestBase [pool-1+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:23:39.735-07:00 INFO DockerTestBase [pool-1+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:23:39.736-07:00 INFO DockerTestBase [pool-1+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:23:39.741-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660CEITD/testRemoteLink/0002_source_thisbuild, network = { name = cbas-docker-test-0001-535c84c3-47e8-4a8f-a96e-cb441ff7606f, id = a5d0aa3c6d635199261eb2920150e046ffa6c25c047a2b84694e504a6f24236f }) 2021-06-09T14:23:39.743-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8091/tcp -> 172.17.0.1:43825/tcp 2021-06-09T14:23:39.744-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8095/tcp -> 172.17.0.1:39044/tcp 2021-06-09T14:23:39.847-07:00 INFO DockerTestBase [pool-1] Starting docker container: 3aab3cdae2bcc 2021-06-09T14:23:39.851-07:00 INFO DockerTestBase [pool-0] Starting docker container: bb5d7bbb50aa6 2021-06-09T14:23:41.880-07:00 INFO DockerTestBase [pool-0] Started docker container: bb5d7bbb50aa6 2021-06-09T14:23:42.059-07:00 INFO DockerTestBase [pool-0] Container information for bb5d7bbb50aa6: host: target.couchbase.host, image: community-6.6.0, ip: 192.168.176.2, 172.17.0.1:37926->192.168.176.2:8091, 172.17.0.1:43609->192.168.176.2:8095 2021-06-09T14:23:42.079-07:00 INFO DockerTestBase [pool-1] Started docker container: 3aab3cdae2bcc 2021-06-09T14:23:42.100-07:00 INFO DockerTestBase [pool-1] Container information for 3aab3cdae2bcc: host: source.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:43825->192.168.176.3:8091, 172.17.0.1:39044->192.168.176.3:8095 2021-06-09T14:23:49.278-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node target (192.168.176.2) 2021-06-09T14:23:49.280-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname target.couchbase.host 2021-06-09T14:23:49.920-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:23:49.946-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node target (192.168.176.2) with services: [DATA] 2021-06-09T14:23:49.946-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:23:50.442-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:23:50.536-07:00 INFO KvStoreHttpHelper [pool-0] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37926] 2021-06-09T14:23:50.542-07:00 INFO KvStoreHttpHelper [pool-0] installing sample(s): [beer-sample] on 172.17.0.1:37926 2021-06-09T14:23:52.643-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node source (192.168.176.3) 2021-06-09T14:23:52.644-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname source.couchbase.host 2021-06-09T14:23:53.733-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:23:53.761-07:00 INFO DockerTestBase [pool-1] Running 'cluster-init' on node source (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:23:53.762-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:23:54.339-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Cluster initialized 2021-06-09T14:23:54.373-07:00 INFO DockerTestBase [pool-1] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:24:07.816-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:43825] 2021-06-09T14:24:07.819-07:00 INFO DockerTestBase [main] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --dataverse Default --name externalcb --type couchbase --hostname target.couchbase.host --link-password couchbase --encryption half --link-username couchbase 2021-06-09T14:24:08.460-07:00 INFO DockerTestBase [main+] >> DEPRECATED: --dataverse is deprecated, please use --scope instead 2021-06-09T14:24:08.461-07:00 INFO DockerTestBase [main+] >> SUCCESS: Link created 2021-06-09T14:24:11.667-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:24:11.670-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:24:12.050-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:24:12.064-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:24:13.065-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:24:13.210-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2053 } 2021-06-09T14:24:13.211-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2053 } 2021-06-09T14:24:14.212-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:24:14.329-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:24:14.330-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:24:14.334-07:00 INFO DockerTestBase [main] Copying files from docker container 3aab3cdae2bcc: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660CEITD/testRemoteLink/0002_source_thisbuild 2021-06-09T14:24:14.334-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container bb5d7bbb50aa6: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660CEITD/testRemoteLink/0001_target_couchbase_community-6.6.0 2021-06-09T14:24:14.688-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: bb5d7bbb50aa6 2021-06-09T14:24:14.754-07:00 INFO DockerTestBase [main] Stopping docker container: 3aab3cdae2bcc 2021-06-09T14:24:15.176-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: bb5d7bbb50aa6 2021-06-09T14:24:15.250-07:00 INFO DockerTestBase [main] Stopped docker container: 3aab3cdae2bcc 2021-06-09T14:24:15.251-07:00 INFO DockerTestBase [main] Removing docker container: bb5d7bbb50aa6 2021-06-09T14:24:15.252-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 3aab3cdae2bcc 2021-06-09T14:24:15.264-07:00 INFO DockerTestBase [main] Removed docker container: bb5d7bbb50aa6 2021-06-09T14:24:15.265-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 3aab3cdae2bcc 2021-06-09T14:24:15.266-07:00 INFO DockerTestBase [main] Removing docker network: a5d0aa3c6d635199261eb2920150e046ffa6c25c047a2b84694e504a6f24236f 2021-06-09T14:24:15.340-07:00 INFO DockerTestBase [main] Removed docker network: a5d0aa3c6d635199261eb2920150e046ffa6c25c047a2b84694e504a6f24236f 2021-06-09T14:24:15.345-07:00 WARN TestMethodTracer [main] ### testRemoteLink SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.648 s - in com.couchbase.analytics.test.docker.remote.CbRemoteLinks660CEITD [INFO] Running com.couchbase.analytics.test.docker.remote.CbRemoteLinks660ITD 2021-06-09 14:24:19,581 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:24:19.661-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:24:19.697-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:24:19.698-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:24:19.782-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.6.0 2021-06-09T14:24:21.067-07:00 INFO DockerTestBase [main+] >> enterprise-6.6.0: Pulling from library/couchbase 2021-06-09T14:24:21.073-07:00 INFO DockerTestBase [main+] >> Digest: sha256:bd26854f1bfe7b5cabe25b342356867d58a07d469a8e52f080519eb05818cf81 2021-06-09T14:24:21.073-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for couchbase:enterprise-6.6.0 2021-06-09T14:24:21.074-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.6.0 2021-06-09T14:24:21.083-07:00 WARN TestMethodTracer [main] ### testLoopbackCliTarget START 2021-06-09T14:24:21.120-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:24:22.287-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-074c7199-f90c-4269-a288-e3db1f2b4e71) 2021-06-09T14:24:22.354-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-074c7199-f90c-4269-a288-e3db1f2b4e71, id = de91eca6a9a8b7e11111dbefd61d87b320b78f556562a95d7dcebc3ccf30cf30 } 2021-06-09T14:24:22.358-07:00 INFO UpgradeTestBase [pool-0] Creating target node ([couchbase:enterprise-6.6.0])... 2021-06-09T14:24:22.359-07:00 INFO UpgradeTestBase [pool-1] Creating source node using 'this build' docker image... 2021-06-09T14:24:22.382-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001target to /opt/couchbase/var 2021-06-09T14:24:22.382-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002source to /opt/couchbase/var 2021-06-09T14:24:22.387-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660ITD/testLoopbackCliTarget/0001_target_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-074c7199-f90c-4269-a288-e3db1f2b4e71, id = de91eca6a9a8b7e11111dbefd61d87b320b78f556562a95d7dcebc3ccf30cf30 }) 2021-06-09T14:24:22.391-07:00 INFO DockerTestBase [pool-1] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:24:22.416-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8091/tcp -> 172.17.0.1:45946/tcp 2021-06-09T14:24:22.417-07:00 INFO DockerTestBase [pool-0] binding exposed port target:8095/tcp -> 172.17.0.1:38252/tcp 2021-06-09T14:24:22.507-07:00 INFO DockerTestBase [pool-1+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:24:22.511-07:00 INFO DockerTestBase [pool-1+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:24:22.511-07:00 INFO DockerTestBase [pool-1+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:24:22.513-07:00 INFO DockerTestBase [pool-1+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:24:22.516-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660ITD/testLoopbackCliTarget/0002_source_thisbuild, network = { name = cbas-docker-test-0001-074c7199-f90c-4269-a288-e3db1f2b4e71, id = de91eca6a9a8b7e11111dbefd61d87b320b78f556562a95d7dcebc3ccf30cf30 }) 2021-06-09T14:24:22.517-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8091/tcp -> 172.17.0.1:43308/tcp 2021-06-09T14:24:22.517-07:00 INFO DockerTestBase [pool-1] binding exposed port source:8095/tcp -> 172.17.0.1:44588/tcp 2021-06-09T14:24:22.553-07:00 INFO DockerTestBase [pool-0] Starting docker container: e088c450e4f02 2021-06-09T14:24:22.602-07:00 INFO DockerTestBase [pool-1] Starting docker container: 5d87de96d097c 2021-06-09T14:24:24.059-07:00 INFO DockerTestBase [pool-1] Started docker container: 5d87de96d097c 2021-06-09T14:24:24.182-07:00 INFO DockerTestBase [pool-1] Container information for 5d87de96d097c: host: source.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:43308->192.168.176.3:8091, 172.17.0.1:44588->192.168.176.3:8095 2021-06-09T14:24:24.227-07:00 INFO DockerTestBase [pool-0] Started docker container: e088c450e4f02 2021-06-09T14:24:24.262-07:00 INFO DockerTestBase [pool-0] Container information for e088c450e4f02: host: target.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.2, 172.17.0.1:45946->192.168.176.2:8091, 172.17.0.1:38252->192.168.176.2:8095 2021-06-09T14:24:31.399-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node target (192.168.176.2) 2021-06-09T14:24:31.401-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname target.couchbase.host 2021-06-09T14:24:32.095-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:24:32.126-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node target (192.168.176.2) with services: [DATA] 2021-06-09T14:24:32.127-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec target.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:24:32.631-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:24:32.694-07:00 INFO KvStoreHttpHelper [pool-0] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:45946] 2021-06-09T14:24:32.698-07:00 INFO KvStoreHttpHelper [pool-0] installing sample(s): [beer-sample] on 172.17.0.1:45946 2021-06-09T14:24:34.388-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node source (192.168.176.3) 2021-06-09T14:24:34.389-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname 127.0.0.1 2021-06-09T14:24:42.047-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:24:42.073-07:00 INFO DockerTestBase [pool-1] Running 'cluster-init' on node source (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:24:42.074-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:24:42.543-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Cluster initialized 2021-06-09T14:24:42.571-07:00 INFO DockerTestBase [pool-1] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:24:56.594-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:43308] 2021-06-09T14:24:56.597-07:00 INFO DockerTestBase [main] Executing the command: docker exec source.couchbase.host /opt/couchbase/bin/couchbase-cli analytics-link-setup -c 192.168.176.3 -u couchbase -p couchbase --create --dataverse Default --name externalcb --type couchbase --hostname target.couchbase.host --link-password couchbase --encryption half --link-username couchbase 2021-06-09T14:24:57.372-07:00 INFO DockerTestBase [main+] >> DEPRECATED: --dataverse is deprecated, please use --scope instead 2021-06-09T14:24:57.372-07:00 INFO DockerTestBase [main+] >> SUCCESS: Link created 2021-06-09T14:25:00.539-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:25:00.541-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:25:01.027-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:25:01.036-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:25:02.037-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:25:02.203-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2943 } 2021-06-09T14:25:02.203-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2943 } 2021-06-09T14:25:03.204-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:25:03.314-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:25:03.314-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:25:03.318-07:00 INFO DockerTestBase [main] Copying files from docker container e088c450e4f02: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660ITD/testLoopbackCliTarget/0001_target_couchbase_enterprise-6.6.0 2021-06-09T14:25:03.318-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 5d87de96d097c: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinks660ITD/testLoopbackCliTarget/0002_source_thisbuild 2021-06-09T14:25:03.649-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 5d87de96d097c 2021-06-09T14:25:03.688-07:00 INFO DockerTestBase [main] Stopping docker container: e088c450e4f02 2021-06-09T14:25:04.186-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 5d87de96d097c 2021-06-09T14:25:04.254-07:00 INFO DockerTestBase [main] Stopped docker container: e088c450e4f02 2021-06-09T14:25:04.256-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: e088c450e4f02 2021-06-09T14:25:04.258-07:00 INFO DockerTestBase [main] Removing docker container: 5d87de96d097c 2021-06-09T14:25:04.280-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: e088c450e4f02 2021-06-09T14:25:04.280-07:00 INFO DockerTestBase [main] Removed docker container: 5d87de96d097c 2021-06-09T14:25:04.282-07:00 INFO DockerTestBase [main] Removing docker network: de91eca6a9a8b7e11111dbefd61d87b320b78f556562a95d7dcebc3ccf30cf30 2021-06-09T14:25:04.365-07:00 INFO DockerTestBase [main] Removed docker network: de91eca6a9a8b7e11111dbefd61d87b320b78f556562a95d7dcebc3ccf30cf30 2021-06-09T14:25:04.373-07:00 WARN TestMethodTracer [main] ### testLoopbackCliTarget SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 47.657 s - in com.couchbase.analytics.test.docker.remote.CbRemoteLinks660ITD [INFO] Running com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD 2021-06-09 14:25:07,985 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:25:08.069-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:25:08.106-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:25:08.106-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:25:08.204-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-5.5.2 2021-06-09T14:25:09.766-07:00 INFO DockerTestBase [main+] >> enterprise-5.5.2: Pulling from library/couchbase 2021-06-09T14:25:10.146-07:00 INFO DockerTestBase [main+] >> 18d680d61657: Pulling fs layer 2021-06-09T14:25:10.146-07:00 INFO DockerTestBase [main+] >> 0addb6fece63: Pulling fs layer 2021-06-09T14:25:10.146-07:00 INFO DockerTestBase [main+] >> 78e58219b215: Pulling fs layer 2021-06-09T14:25:10.146-07:00 INFO DockerTestBase [main+] >> eb6959a66df2: Pulling fs layer 2021-06-09T14:25:10.147-07:00 INFO DockerTestBase [main+] >> 1b11929e4bba: Pulling fs layer 2021-06-09T14:25:10.147-07:00 INFO DockerTestBase [main+] >> a5c53dfb80d7: Pulling fs layer 2021-06-09T14:25:10.147-07:00 INFO DockerTestBase [main+] >> f8ba4b377591: Pulling fs layer 2021-06-09T14:25:10.147-07:00 INFO DockerTestBase [main+] >> 18924b37f297: Pulling fs layer 2021-06-09T14:25:10.148-07:00 INFO DockerTestBase [main+] >> 344e83dbe3d3: Pulling fs layer 2021-06-09T14:25:10.148-07:00 INFO DockerTestBase [main+] >> 1049a208a670: Pulling fs layer 2021-06-09T14:25:10.148-07:00 INFO DockerTestBase [main+] >> 36612a3dd21b: Pulling fs layer 2021-06-09T14:25:10.148-07:00 INFO DockerTestBase [main+] >> 6fa79e1ff301: Pulling fs layer 2021-06-09T14:25:10.148-07:00 INFO DockerTestBase [main+] >> 1801915e2460: Pulling fs layer 2021-06-09T14:25:10.149-07:00 INFO DockerTestBase [main+] >> 18924b37f297: Waiting 2021-06-09T14:25:10.149-07:00 INFO DockerTestBase [main+] >> 344e83dbe3d3: Waiting 2021-06-09T14:25:10.149-07:00 INFO DockerTestBase [main+] >> 1049a208a670: Waiting 2021-06-09T14:25:10.149-07:00 INFO DockerTestBase [main+] >> 36612a3dd21b: Waiting 2021-06-09T14:25:10.149-07:00 INFO DockerTestBase [main+] >> 6fa79e1ff301: Waiting 2021-06-09T14:25:10.150-07:00 INFO DockerTestBase [main+] >> 1801915e2460: Waiting 2021-06-09T14:25:10.150-07:00 INFO DockerTestBase [main+] >> eb6959a66df2: Waiting 2021-06-09T14:25:10.150-07:00 INFO DockerTestBase [main+] >> 1b11929e4bba: Waiting 2021-06-09T14:25:10.150-07:00 INFO DockerTestBase [main+] >> a5c53dfb80d7: Waiting 2021-06-09T14:25:10.150-07:00 INFO DockerTestBase [main+] >> f8ba4b377591: Waiting 2021-06-09T14:25:10.484-07:00 INFO DockerTestBase [main+] >> 0addb6fece63: Verifying Checksum 2021-06-09T14:25:10.485-07:00 INFO DockerTestBase [main+] >> 0addb6fece63: Download complete 2021-06-09T14:25:10.511-07:00 INFO DockerTestBase [main+] >> 78e58219b215: Verifying Checksum 2021-06-09T14:25:10.512-07:00 INFO DockerTestBase [main+] >> 78e58219b215: Download complete 2021-06-09T14:25:10.878-07:00 INFO DockerTestBase [main+] >> eb6959a66df2: Verifying Checksum 2021-06-09T14:25:10.878-07:00 INFO DockerTestBase [main+] >> eb6959a66df2: Download complete 2021-06-09T14:25:11.151-07:00 INFO DockerTestBase [main+] >> 1b11929e4bba: Verifying Checksum 2021-06-09T14:25:11.151-07:00 INFO DockerTestBase [main+] >> 1b11929e4bba: Download complete 2021-06-09T14:25:11.180-07:00 INFO DockerTestBase [main+] >> 18d680d61657: Verifying Checksum 2021-06-09T14:25:11.180-07:00 INFO DockerTestBase [main+] >> 18d680d61657: Download complete 2021-06-09T14:25:11.275-07:00 INFO DockerTestBase [main+] >> a5c53dfb80d7: Verifying Checksum 2021-06-09T14:25:11.276-07:00 INFO DockerTestBase [main+] >> a5c53dfb80d7: Download complete 2021-06-09T14:25:11.538-07:00 INFO DockerTestBase [main+] >> 18924b37f297: Verifying Checksum 2021-06-09T14:25:11.538-07:00 INFO DockerTestBase [main+] >> 18924b37f297: Download complete 2021-06-09T14:25:11.630-07:00 INFO DockerTestBase [main+] >> 344e83dbe3d3: Verifying Checksum 2021-06-09T14:25:11.631-07:00 INFO DockerTestBase [main+] >> 344e83dbe3d3: Download complete 2021-06-09T14:25:11.887-07:00 INFO DockerTestBase [main+] >> 1049a208a670: Verifying Checksum 2021-06-09T14:25:11.888-07:00 INFO DockerTestBase [main+] >> 1049a208a670: Download complete 2021-06-09T14:25:11.990-07:00 INFO DockerTestBase [main+] >> 36612a3dd21b: Verifying Checksum 2021-06-09T14:25:11.990-07:00 INFO DockerTestBase [main+] >> 36612a3dd21b: Download complete 2021-06-09T14:25:12.256-07:00 INFO DockerTestBase [main+] >> 6fa79e1ff301: Verifying Checksum 2021-06-09T14:25:12.257-07:00 INFO DockerTestBase [main+] >> 6fa79e1ff301: Download complete 2021-06-09T14:25:12.352-07:00 INFO DockerTestBase [main+] >> 1801915e2460: Verifying Checksum 2021-06-09T14:25:12.353-07:00 INFO DockerTestBase [main+] >> 1801915e2460: Download complete 2021-06-09T14:25:13.520-07:00 INFO DockerTestBase [main+] >> 18d680d61657: Pull complete 2021-06-09T14:25:13.661-07:00 INFO DockerTestBase [main+] >> 0addb6fece63: Pull complete 2021-06-09T14:25:13.844-07:00 INFO DockerTestBase [main+] >> 78e58219b215: Pull complete 2021-06-09T14:25:14.099-07:00 INFO DockerTestBase [main+] >> eb6959a66df2: Pull complete 2021-06-09T14:25:15.036-07:00 INFO DockerTestBase [main+] >> f8ba4b377591: Verifying Checksum 2021-06-09T14:25:15.037-07:00 INFO DockerTestBase [main+] >> f8ba4b377591: Download complete 2021-06-09T14:25:15.339-07:00 INFO DockerTestBase [main+] >> 1b11929e4bba: Pull complete 2021-06-09T14:25:15.448-07:00 INFO DockerTestBase [main+] >> a5c53dfb80d7: Pull complete 2021-06-09T14:25:23.122-07:00 INFO DockerTestBase [main+] >> f8ba4b377591: Pull complete 2021-06-09T14:25:23.603-07:00 INFO DockerTestBase [main+] >> 18924b37f297: Pull complete 2021-06-09T14:25:23.963-07:00 INFO DockerTestBase [main+] >> 344e83dbe3d3: Pull complete 2021-06-09T14:25:24.328-07:00 INFO DockerTestBase [main+] >> 1049a208a670: Pull complete 2021-06-09T14:25:24.603-07:00 INFO DockerTestBase [main+] >> 36612a3dd21b: Pull complete 2021-06-09T14:25:24.995-07:00 INFO DockerTestBase [main+] >> 6fa79e1ff301: Pull complete 2021-06-09T14:25:25.156-07:00 INFO DockerTestBase [main+] >> 1801915e2460: Pull complete 2021-06-09T14:25:25.163-07:00 INFO DockerTestBase [main+] >> Digest: sha256:dcdf07a219ba5fe10bf6270082eb57b60320bb520a1f019b4793b9d0ec9fa2ba 2021-06-09T14:25:25.165-07:00 INFO DockerTestBase [main+] >> Status: Downloaded newer image for couchbase:enterprise-5.5.2 2021-06-09T14:25:25.167-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-5.5.2 2021-06-09T14:25:25.177-07:00 WARN TestMethodTracer [main] ### testPreAlice START 2021-06-09T14:25:25.214-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:25:26.504-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-8bfe5c4c-eaab-45f3-a15f-e415ee4fb61a) 2021-06-09T14:25:26.577-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-8bfe5c4c-eaab-45f3-a15f-e415ee4fb61a, id = a0a5526c298635f456ae5b5ba9b421538e585dfdfe0e90169aa0a471d0489aac } 2021-06-09T14:25:26.584-07:00 INFO UpgradeTestBase [pool-0] Creating n1 node ([couchbase:enterprise-5.5.2])... 2021-06-09T14:25:26.600-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:25:26.608-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD/testPreAlice/0001_n1_couchbase_enterprise-5.5.2, network = { name = cbas-docker-test-0001-8bfe5c4c-eaab-45f3-a15f-e415ee4fb61a, id = a0a5526c298635f456ae5b5ba9b421538e585dfdfe0e90169aa0a471d0489aac }) 2021-06-09T14:25:26.615-07:00 INFO UpgradeTestBase [pool-1] Creating n2 node using 'this build' docker image... 2021-06-09T14:25:26.616-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:25:26.616-07:00 INFO DockerTestBase [pool-1] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:25:26.645-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8091/tcp -> 172.17.0.1:36238/tcp 2021-06-09T14:25:26.646-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8095/tcp -> 172.17.0.1:34019/tcp 2021-06-09T14:25:26.739-07:00 INFO DockerTestBase [pool-1+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:25:26.740-07:00 INFO DockerTestBase [pool-1+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:25:26.741-07:00 INFO DockerTestBase [pool-1+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:25:26.743-07:00 INFO DockerTestBase [pool-1+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:25:26.745-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD/testPreAlice/0002_n2_thisbuild, network = { name = cbas-docker-test-0001-8bfe5c4c-eaab-45f3-a15f-e415ee4fb61a, id = a0a5526c298635f456ae5b5ba9b421538e585dfdfe0e90169aa0a471d0489aac }) 2021-06-09T14:25:26.747-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8091/tcp -> 172.17.0.1:43055/tcp 2021-06-09T14:25:26.747-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8095/tcp -> 172.17.0.1:45376/tcp 2021-06-09T14:25:26.810-07:00 INFO DockerTestBase [pool-0] Starting docker container: 5c00dd764fd7d 2021-06-09T14:25:26.827-07:00 INFO DockerTestBase [pool-1] Starting docker container: 11d152169bd28 2021-06-09T14:25:29.211-07:00 INFO DockerTestBase [pool-1] Started docker container: 11d152169bd28 2021-06-09T14:25:29.271-07:00 INFO DockerTestBase [pool-0] Started docker container: 5c00dd764fd7d 2021-06-09T14:25:29.524-07:00 INFO DockerTestBase [pool-1] Container information for 11d152169bd28: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:43055->192.168.176.3:8091, 172.17.0.1:45376->192.168.176.3:8095 2021-06-09T14:25:29.527-07:00 INFO DockerTestBase [pool-0] Container information for 5c00dd764fd7d: host: n1.couchbase.host, image: enterprise-5.5.2, ip: 192.168.176.2, 172.17.0.1:36238->192.168.176.2:8091, 172.17.0.1:34019->192.168.176.2:8095 2021-06-09T14:25:39.737-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:25:39.739-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:25:41.493-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:25:41.530-07:00 INFO DockerTestBase [pool-1] Running 'cluster-init' on node n2 (192.168.176.3) with services: [DATA, ANALYTICS] 2021-06-09T14:25:41.530-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:25:42.189-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Cluster initialized 2021-06-09T14:25:42.851-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:25:42.851-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:25:43.338-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:25:43.348-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:25:43.348-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:25:45.414-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:25:45.434-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:25:59.458-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36238] 2021-06-09T14:25:59.464-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:36238 2021-06-09T14:25:59.507-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:43055] 2021-06-09T14:26:00.559-07:00 INFO CbRemoteLinksIllegalVersionITD [main] create link threw exception java.lang.Exception: Invalid cluster compatibility version for remote link Default.externalcb: must be >= 6.0.x, but was 5.5.x (in line 1, at column 1) at org.apache.asterix.test.common.ResultExtractor.checkForErrors(ResultExtractor.java:305) ~[asterix-app-0.9.6-SNAPSHOT-tests.jar:0.9.6-SNAPSHOT] at org.apache.asterix.test.common.ResultExtractor.extract(ResultExtractor.java:220) ~[asterix-app-0.9.6-SNAPSHOT-tests.jar:0.9.6-SNAPSHOT] at org.apache.asterix.test.common.ResultExtractor.extract(ResultExtractor.java:155) ~[asterix-app-0.9.6-SNAPSHOT-tests.jar:0.9.6-SNAPSHOT] at org.apache.asterix.test.common.TestExecutor.executeUpdateOrDdl(TestExecutor.java:1765) ~[asterix-app-0.9.6-SNAPSHOT-tests.jar:0.9.6-SNAPSHOT] at org.apache.asterix.test.common.TestExecutor.executeSqlppUpdateOrDdl(TestExecutor.java:1754) ~[asterix-app-0.9.6-SNAPSHOT-tests.jar:0.9.6-SNAPSHOT] at com.couchbase.analytics.test.common.ClusterTestBase.executeSqlppUpdateOrDdl(ClusterTestBase.java:1336) ~[cbas-server-7.0.0-SNAPSHOT-tests.jar:] at com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD.testIllegalTargetVersion(CbRemoteLinksIllegalVersionITD.java:60) [test-classes/:?] at com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD.testPreAlice(CbRemoteLinksIllegalVersionITD.java:36) [test-classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) [junit-4.13.jar:4.13] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.13.jar:4.13] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.jar:4.13] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) [surefire-booter-2.21.0.jar:2.21.0] 2021-06-09T14:26:00.578-07:00 INFO DockerTestBase [main] Copying files from docker container 5c00dd764fd7d: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD/testPreAlice/0001_n1_couchbase_enterprise-5.5.2 2021-06-09T14:26:00.579-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 11d152169bd28: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD/testPreAlice/0002_n2_thisbuild 2021-06-09T14:26:00.830-07:00 INFO DockerTestBase [main] Stopping docker container: 5c00dd764fd7d 2021-06-09T14:26:00.897-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 11d152169bd28 2021-06-09T14:26:01.306-07:00 INFO DockerTestBase [main] Stopped docker container: 5c00dd764fd7d 2021-06-09T14:26:01.498-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 11d152169bd28 2021-06-09T14:26:01.500-07:00 INFO DockerTestBase [main] Removing docker container: 11d152169bd28 2021-06-09T14:26:01.501-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 5c00dd764fd7d 2021-06-09T14:26:01.579-07:00 INFO DockerTestBase [main] Removed docker container: 11d152169bd28 2021-06-09T14:26:01.656-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 5c00dd764fd7d 2021-06-09T14:26:01.657-07:00 INFO DockerTestBase [main] Removing docker network: a0a5526c298635f456ae5b5ba9b421538e585dfdfe0e90169aa0a471d0489aac 2021-06-09T14:26:01.835-07:00 INFO DockerTestBase [main] Removed docker network: a0a5526c298635f456ae5b5ba9b421538e585dfdfe0e90169aa0a471d0489aac 2021-06-09T14:26:01.843-07:00 WARN TestMethodTracer [main] ### testPreAlice SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.398 s - in com.couchbase.analytics.test.docker.remote.CbRemoteLinksIllegalVersionITD [INFO] Running com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD 2021-06-09 14:26:04,420 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:26:04.506-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:26:04.547-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:26:04.547-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:26:04.596-07:00 WARN TestMethodTracer [main] ### testRemoteCluster START 2021-06-09T14:26:04.635-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:26:05.848-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-56afa16e-3c34-4d0b-81f3-f22d6025d351) 2021-06-09T14:26:05.923-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-56afa16e-3c34-4d0b-81f3-f22d6025d351, id = 353d470c026987a8d5fdd5c409020a5662ff919d4a121d069bcff8e895d26713 } 2021-06-09T14:26:05.930-07:00 INFO UpgradeTestBase [pool-1] Creating n2 node using 'this build' docker image... 2021-06-09T14:26:05.936-07:00 INFO UpgradeTestBase [pool-0] Creating n1 node using 'this build' docker image... 2021-06-09T14:26:05.956-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n2 to /opt/couchbase/var 2021-06-09T14:26:05.956-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n1 to /opt/couchbase/var 2021-06-09T14:26:05.969-07:00 INFO DockerTestBase [pool-1] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:26:06.066-07:00 INFO DockerTestBase [pool-1+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:26:06.066-07:00 INFO DockerTestBase [pool-1+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:26:06.066-07:00 INFO DockerTestBase [pool-1+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:26:06.068-07:00 INFO DockerTestBase [pool-1+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:26:06.074-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD/testRemoteCluster/0002_n1_thisbuild, network = { name = cbas-docker-test-0001-56afa16e-3c34-4d0b-81f3-f22d6025d351, id = 353d470c026987a8d5fdd5c409020a5662ff919d4a121d069bcff8e895d26713 }) 2021-06-09T14:26:06.074-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD/testRemoteCluster/0001_n2_thisbuild, network = { name = cbas-docker-test-0001-56afa16e-3c34-4d0b-81f3-f22d6025d351, id = 353d470c026987a8d5fdd5c409020a5662ff919d4a121d069bcff8e895d26713 }) 2021-06-09T14:26:06.115-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8091/tcp -> 172.17.0.1:39733/tcp 2021-06-09T14:26:06.116-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8091/tcp -> 172.17.0.1:37059/tcp 2021-06-09T14:26:06.116-07:00 INFO DockerTestBase [pool-0] binding exposed port n1:8095/tcp -> 172.17.0.1:46798/tcp 2021-06-09T14:26:06.117-07:00 INFO DockerTestBase [pool-1] binding exposed port n2:8095/tcp -> 172.17.0.1:37878/tcp 2021-06-09T14:26:06.292-07:00 INFO DockerTestBase [pool-1] Starting docker container: ef58a07a833cc 2021-06-09T14:26:06.292-07:00 INFO DockerTestBase [pool-0] Starting docker container: b5ea8adb7463b 2021-06-09T14:26:07.299-07:00 INFO DockerTestBase [pool-1] Started docker container: ef58a07a833cc 2021-06-09T14:26:07.399-07:00 INFO DockerTestBase [pool-0] Started docker container: b5ea8adb7463b 2021-06-09T14:26:07.497-07:00 INFO DockerTestBase [pool-1] Container information for ef58a07a833cc: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:37878->192.168.176.2:8095, 172.17.0.1:37059->192.168.176.2:8091 2021-06-09T14:26:07.499-07:00 INFO DockerTestBase [pool-0] Container information for b5ea8adb7463b: host: n1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:46798->192.168.176.3:8095, 172.17.0.1:39733->192.168.176.3:8091 2021-06-09T14:26:17.702-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1 (192.168.176.3) 2021-06-09T14:26:17.705-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:26:17.783-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node n2 (192.168.176.2) 2021-06-09T14:26:17.784-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:26:18.614-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:26:18.654-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node n1 (192.168.176.3) with services: [DATA] 2021-06-09T14:26:18.655-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:26:18.996-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:26:19.022-07:00 INFO DockerTestBase [pool-1] Running 'cluster-init' on node n2 (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T14:26:19.022-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T14:26:19.170-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:26:19.505-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Cluster initialized 2021-06-09T14:26:19.550-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:26:32.111-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:39733] 2021-06-09T14:26:32.115-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:39733 2021-06-09T14:26:33.194-07:00 INFO DockerTestBase [main] Generating certificates for cluster 1 2021-06-09T14:26:33.195-07:00 INFO DockerTestBase [main] openssl commands will be logged to openssl.log 2021-06-09T14:26:33.196-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/ca.key 2048 2021-06-09T14:26:33.379-07:00 INFO DockerTestBase [main] Executing openssl command: req -config /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/rootConfig.conf -new -x509 -days 1 -sha256 -key cluster1/private/ca.key -out cluster1/public/ca.pem -subj /C=UA/O=MyCompany/CN=RootCA 2021-06-09T14:26:33.389-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/public/ca.pem 2021-06-09T14:26:33.405-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/int.key 1024 2021-06-09T14:26:33.446-07:00 INFO DockerTestBase [main] Executing openssl command: req -new -days 1 -sha256 -key cluster1/private/int.key -out cluster1/requests/server-signing.csr -subj /C=UA/O=MyCompany/OU=Servers/CN=ServerSigningCA 2021-06-09T14:26:33.455-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -CA cluster1/public/ca.pem -CAkey cluster1/private/ca.key -CAcreateserial -CAserial cluster1/public/server-signing.srl -days 1 -req -in cluster1/requests/server-signing.csr -out cluster1/public/int.pem -extfile /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/int.ext 2021-06-09T14:26:33.466-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/public/int.pem 2021-06-09T14:26:33.473-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/private/couchbase.default.svc.n_1.key 2048 2021-06-09T14:26:33.574-07:00 INFO DockerTestBase [main] Executing openssl command: req -new -sha256 -key cluster1/private/couchbase.default.svc.n_1.key -out cluster1/requests/couchbase.default.svc.n_1.csr -subj /CN=Couchbase Server 2021-06-09T14:26:33.583-07:00 INFO DockerTestBase [main] Executing openssl command: req -text -noout -verify -in cluster1/requests/couchbase.default.svc.n_1.csr 2021-06-09T14:26:33.590-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -CA cluster1/public/int.pem -CAkey cluster1/private/int.key -CAcreateserial -days 1 -req -sha256 -in cluster1/requests/couchbase.default.svc.n_1.csr -out cluster1/public/couchbase.default.svc.n_1.pem -extfile cluster1/server.ext 2021-06-09T14:26:33.602-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/public/couchbase.default.svc.n_1.pem 2021-06-09T14:26:33.615-07:00 INFO DockerTestBase [main] +http://172.17.0.1:39733/controller/uploadClusterCA 2021-06-09T14:26:33.701-07:00 INFO DockerTestBase [main] Executing the command: bash -c chmod a+x /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n1/lib/couchbase/inbox/chain.pem 2021-06-09T14:26:33.708-07:00 INFO DockerTestBase [main] Executing the command: bash -c chmod a+x /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n1/lib/couchbase/inbox/pkey.key 2021-06-09T14:26:33.714-07:00 INFO DockerTestBase [main] +http://172.17.0.1:39733/node/controller/reloadCertificate 2021-06-09T14:26:33.793-07:00 INFO DockerTestBase [main] +http://172.17.0.1:39733/settings/clientCertAuth 2021-06-09T14:26:33.807-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37059] 2021-06-09T14:26:33.812-07:00 INFO DockerTestBase [main] Executing openssl command: genrsa -out cluster1/client/client_couchbase.key 2048 2021-06-09T14:26:33.919-07:00 INFO DockerTestBase [main] Executing openssl command: req -new -key cluster1/client/client_couchbase.key -out cluster1/client/client_couchbase.csr -subj /CN=couchbase 2021-06-09T14:26:33.928-07:00 INFO DockerTestBase [main] Executing openssl command: req -text -noout -verify -in cluster1/client/client_couchbase.csr 2021-06-09T14:26:33.935-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -CA cluster1/public/int.pem -CAkey cluster1/private/int.key -CAcreateserial -days 1 -req -in cluster1/client/client_couchbase.csr -out cluster1/client/client_couchbase.pem -extfile /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-server/target/test-classes/tls/client.ext 2021-06-09T14:26:33.943-07:00 INFO DockerTestBase [main] Executing openssl command: x509 -text -noout -in cluster1/client/client_couchbasechain.pem 2021-06-09T14:26:40.116-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:26:40.119-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:26:40.445-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:26:40.450-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:26:41.452-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:26:41.579-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 466 } 2021-06-09T14:26:41.580-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 466 } 2021-06-09T14:26:42.581-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:26:42.824-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5179 } 2021-06-09T14:26:42.824-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 5179 } 2021-06-09T14:26:43.938-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:26:43.938-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:26:43.942-07:00 INFO DockerTestBase [main] Copying files from docker container b5ea8adb7463b: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD/testRemoteCluster/0002_n1_thisbuild 2021-06-09T14:26:43.942-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container ef58a07a833cc: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD/testRemoteCluster/0001_n2_thisbuild 2021-06-09T14:26:44.290-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: ef58a07a833cc 2021-06-09T14:26:44.323-07:00 INFO DockerTestBase [main] Stopping docker container: b5ea8adb7463b 2021-06-09T14:26:44.867-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: ef58a07a833cc 2021-06-09T14:26:44.948-07:00 INFO DockerTestBase [main] Stopped docker container: b5ea8adb7463b 2021-06-09T14:26:44.949-07:00 INFO DockerTestBase [main] Removing docker container: ef58a07a833cc 2021-06-09T14:26:44.951-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: b5ea8adb7463b 2021-06-09T14:26:44.962-07:00 INFO DockerTestBase [main] Removed docker container: ef58a07a833cc 2021-06-09T14:26:44.964-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: b5ea8adb7463b 2021-06-09T14:26:44.966-07:00 INFO DockerTestBase [main] Removing docker network: 353d470c026987a8d5fdd5c409020a5662ff919d4a121d069bcff8e895d26713 2021-06-09T14:26:45.044-07:00 INFO DockerTestBase [main] Removed docker network: 353d470c026987a8d5fdd5c409020a5662ff919d4a121d069bcff8e895d26713 2021-06-09T14:26:45.054-07:00 WARN TestMethodTracer [main] ### testRemoteCluster SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.338 s - in com.couchbase.analytics.test.docker.remote.CbRemoteTLSCertificatesITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD 2021-06-09 14:26:47,793 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:26:47.881-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:26:47.934-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:26:47.935-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:26:48.033-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.0.3 2021-06-09T14:26:49.086-07:00 INFO DockerTestBase [main+] >> enterprise-6.0.3: Pulling from library/couchbase 2021-06-09T14:26:49.091-07:00 INFO DockerTestBase [main+] >> Digest: sha256:61e4543f38022a6e76683a5002536243e2af53e75fa23a72613f828100657b4d 2021-06-09T14:26:49.091-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for couchbase:enterprise-6.0.3 2021-06-09T14:26:49.093-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.0.3 2021-06-09T14:26:49.102-07:00 WARN TestMethodTracer [main] ### upgradeVerifyClusterCompat START 2021-06-09T14:26:49.142-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:26:50.427-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-4954bf49-e09c-4db9-b722-ac9f1a7d069e) 2021-06-09T14:26:50.506-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-4954bf49-e09c-4db9-b722-ac9f1a7d069e, id = 1cd3c07b468b7bf322657f22676ddf0c97997e5400fa40331eb409eaad36d89c } 2021-06-09T14:26:50.508-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.0.3])... 2021-06-09T14:26:50.511-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:26:50.516-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0001_n1_couchbase_enterprise-6.0.3, network = { name = cbas-docker-test-0001-4954bf49-e09c-4db9-b722-ac9f1a7d069e, id = 1cd3c07b468b7bf322657f22676ddf0c97997e5400fa40331eb409eaad36d89c }) 2021-06-09T14:26:50.548-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:34142/tcp 2021-06-09T14:26:50.549-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:33560/tcp 2021-06-09T14:26:50.714-07:00 INFO DockerTestBase [main] Starting docker container: b4f5381294069 2021-06-09T14:26:51.777-07:00 INFO DockerTestBase [main] Started docker container: b4f5381294069 2021-06-09T14:26:51.912-07:00 INFO DockerTestBase [main] Container information for b4f5381294069: host: n1.couchbase.host, image: enterprise-6.0.3, ip: 192.168.176.2, 172.17.0.1:34142->192.168.176.2:8091, 172.17.0.1:33560->192.168.176.2:8095 2021-06-09T14:27:03.030-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:27:03.032-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:27:03.590-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:27:03.598-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.0.3])... 2021-06-09T14:27:03.599-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:27:03.599-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0002_n2_couchbase_enterprise-6.0.3, network = { name = cbas-docker-test-0001-4954bf49-e09c-4db9-b722-ac9f1a7d069e, id = 1cd3c07b468b7bf322657f22676ddf0c97997e5400fa40331eb409eaad36d89c }) 2021-06-09T14:27:03.600-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:34777/tcp 2021-06-09T14:27:03.600-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:45104/tcp 2021-06-09T14:27:03.663-07:00 INFO DockerTestBase [main] Starting docker container: 0825af40df51f 2021-06-09T14:27:04.424-07:00 INFO DockerTestBase [main] Started docker container: 0825af40df51f 2021-06-09T14:27:04.434-07:00 INFO DockerTestBase [main] Container information for 0825af40df51f: host: n2.couchbase.host, image: enterprise-6.0.3, ip: 192.168.176.3, 172.17.0.1:34777->192.168.176.3:8091, 172.17.0.1:45104->192.168.176.3:8095 2021-06-09T14:27:14.644-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:27:14.645-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:27:15.248-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:27:15.260-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:27:15.261-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:27:15.627-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:27:15.637-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:27:15.638-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:27:18.903-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:27:18.910-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:27:18.911-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:27:20.255-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:27:20.255-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:27:20.282-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:27:20.298-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:27:20.299-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:27:20.308-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:27:27.760-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:34142] 2021-06-09T14:27:27.765-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:34142 2021-06-09T14:27:27.824-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:34777] 2021-06-09T14:27:31.610-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:27:31.612-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:27:31.931-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:27:31.935-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:27:32.936-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:27:33.139-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 833 } 2021-06-09T14:27:33.139-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 833 } 2021-06-09T14:27:34.141-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T14:27:34.299-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:27:34.300-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:27:34.301-07:00 INFO UpgradeTestBase [main] Upgrading [n2.couchbase.host] (online, rebalance) ... 2021-06-09T14:27:34.304-07:00 INFO UpgradeTestBase [pool-0] Creating n2-new node using 'this build' docker image... 2021-06-09T14:27:34.304-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n2new to /opt/couchbase/var 2021-06-09T14:27:34.305-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:27:34.399-07:00 INFO DockerTestBase [pool-0+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:27:34.400-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:27:34.400-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:27:34.401-07:00 INFO DockerTestBase [pool-0+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:27:34.404-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0003_n2new_thisbuild, network = { name = cbas-docker-test-0001-4954bf49-e09c-4db9-b722-ac9f1a7d069e, id = 1cd3c07b468b7bf322657f22676ddf0c97997e5400fa40331eb409eaad36d89c }) 2021-06-09T14:27:34.405-07:00 INFO DockerTestBase [pool-0] binding exposed port n2new:8091/tcp -> 172.17.0.1:37832/tcp 2021-06-09T14:27:34.405-07:00 INFO DockerTestBase [pool-0] binding exposed port n2new:8095/tcp -> 172.17.0.1:39004/tcp 2021-06-09T14:27:34.464-07:00 INFO DockerTestBase [pool-0] Starting docker container: 5f5d00c9e7be8 2021-06-09T14:27:35.145-07:00 INFO DockerTestBase [pool-0] Started docker container: 5f5d00c9e7be8 2021-06-09T14:27:35.154-07:00 INFO DockerTestBase [pool-0] Container information for 5f5d00c9e7be8: host: n2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:37832->192.168.176.4:8091, 172.17.0.1:39004->192.168.176.4:8095 2021-06-09T14:27:45.204-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n2new (192.168.176.4) 2021-06-09T14:27:45.204-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n2new.couchbase.host 2021-06-09T14:27:46.785-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:27:46.807-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node n2new (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:27:46.807-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:27:50.524-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T14:27:50.532-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T14:27:50.533-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:27:50.534-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase --server-remove n2.couchbase.host 2021-06-09T14:28:10.770-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:28:10.770-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:28:10.777-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:28:10.795-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:28:10.796-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:28:10.799-07:00 INFO DockerTestBase [main] Copying files from docker container 0825af40df51f: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0002_n2_couchbase_enterprise-6.0.3 2021-06-09T14:28:11.042-07:00 INFO DockerTestBase [main] Stopping docker container: 0825af40df51f 2021-06-09T14:28:11.462-07:00 INFO DockerTestBase [main] Stopped docker container: 0825af40df51f 2021-06-09T14:28:11.463-07:00 INFO DockerTestBase [main] Removing docker container: 0825af40df51f 2021-06-09T14:28:11.480-07:00 INFO DockerTestBase [main] Removed docker container: 0825af40df51f 2021-06-09T14:28:11.482-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:28:19.057-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37832] 2021-06-09T14:28:19.058-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:28:19.620-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:28:19.620-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:28:19.845-07:00 INFO DockerTestBase [main] assuming 6.0 compatibility on missing 'clusterCompatibility' field 2021-06-09T14:28:19.846-07:00 INFO UpgradeTestBase [main] Upgrading [n1.couchbase.host] (online, rebalance) ... 2021-06-09T14:28:19.849-07:00 INFO UpgradeTestBase [pool-0] Creating n1-new node using 'this build' docker image... 2021-06-09T14:28:19.849-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n1new to /opt/couchbase/var 2021-06-09T14:28:19.850-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0004_n1new_thisbuild, network = { name = cbas-docker-test-0001-4954bf49-e09c-4db9-b722-ac9f1a7d069e, id = 1cd3c07b468b7bf322657f22676ddf0c97997e5400fa40331eb409eaad36d89c }) 2021-06-09T14:28:19.850-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8091/tcp -> 172.17.0.1:46026/tcp 2021-06-09T14:28:19.850-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8095/tcp -> 172.17.0.1:38937/tcp 2021-06-09T14:28:19.916-07:00 INFO DockerTestBase [pool-0] Starting docker container: debce191d355e 2021-06-09T14:28:20.741-07:00 INFO DockerTestBase [pool-0] Started docker container: debce191d355e 2021-06-09T14:28:20.747-07:00 INFO DockerTestBase [pool-0] Container information for debce191d355e: host: n1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:46026->192.168.176.3:8091, 172.17.0.1:38937->192.168.176.3:8095 2021-06-09T14:28:30.788-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1new (192.168.176.3) 2021-06-09T14:28:30.788-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n1new.couchbase.host 2021-06-09T14:28:31.860-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:28:31.886-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node n1new (192.168.176.3) with services: [DATA] 2021-06-09T14:28:31.887-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://n1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T14:28:35.855-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T14:28:35.883-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T14:28:35.884-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:28:35.884-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove n1.couchbase.host 2021-06-09T14:32:43.793-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:32:43.794-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:32:43.844-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:32:43.855-07:00 INFO DockerTestBase [pool-2] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:32:43.856-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:32:43.856-07:00 INFO DockerTestBase [main] Copying files from docker container b4f5381294069: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0001_n1_couchbase_enterprise-6.0.3 2021-06-09T14:32:46.546-07:00 INFO DockerTestBase [main] Stopping docker container: b4f5381294069 2021-06-09T14:32:46.960-07:00 INFO DockerTestBase [main] Stopped docker container: b4f5381294069 2021-06-09T14:32:46.961-07:00 INFO DockerTestBase [main] Removing docker container: b4f5381294069 2021-06-09T14:32:46.973-07:00 INFO DockerTestBase [main] Removed docker container: b4f5381294069 2021-06-09T14:32:46.978-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T14:32:46.999-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@797bb429[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@7d443f59] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:48.014-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3e234afd[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@30e4a291] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:49.025-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5ffd4180[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@513ab5d7] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:50.041-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@21400ac5[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@47a8692b] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:51.054-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@8bf766c[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@67898d3] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:52.066-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7f973eb8[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@7150cee2] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:53.078-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@70302739[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@7ef7d697] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:54.089-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@12e439d[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d1c40@fe9bc10] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@30dea8e8[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 42] 2021-06-09T14:32:55.094-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:32:56.098-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:32:57.103-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:32:58.107-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:32:59.112-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:33:00.120-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:33:01.126-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:33:02.139-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:33:04.318-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:33:05.934-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37832] 2021-06-09T14:33:05.934-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:33:05.935-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T14:33:06.758-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:33:06.758-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:33:06.758-07:00 INFO UpgradeTestBase [main] Verifying metadata 2021-06-09T14:33:06.759-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37832] 2021-06-09T14:33:06.760-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T14:33:07.196-07:00 INFO UpgradeFrom603ITD [main] topology nodes: [NodeInfo{nodeId='9e87c63af49fb8245231a0dbff9e81d7', priority=1970324837040384, opaque={cbas-version=7.0.1-1541690320, cc-http-port=9111, controller-id=1, host=n2new.couchbase.host, ns-server-port=8091, num-iodevices=1, starting-partition-id=1, svc-http-port=8095}}] 2021-06-09T14:33:07.218-07:00 INFO UpgradeFrom603ITD [main] 9e87c63af49fb8245231a0dbff9e81d7 has cbas version ProductVersion 7.0.1 2021-06-09T14:33:07.225-07:00 INFO DockerTestBase [main] Copying files from docker container debce191d355e: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0004_n1new_thisbuild 2021-06-09T14:33:07.233-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 5f5d00c9e7be8: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD/upgradeVerifyClusterCompat/0003_n2new_thisbuild 2021-06-09T14:33:08.433-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 5f5d00c9e7be8 2021-06-09T14:33:09.078-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 5f5d00c9e7be8 2021-06-09T14:33:10.185-07:00 INFO DockerTestBase [main] Stopping docker container: debce191d355e 2021-06-09T14:33:10.558-07:00 INFO DockerTestBase [main] Stopped docker container: debce191d355e 2021-06-09T14:33:10.559-07:00 INFO DockerTestBase [main] Removing docker container: debce191d355e 2021-06-09T14:33:10.560-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 5f5d00c9e7be8 2021-06-09T14:33:10.568-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 5f5d00c9e7be8 2021-06-09T14:33:10.570-07:00 INFO DockerTestBase [main] Removed docker container: debce191d355e 2021-06-09T14:33:10.571-07:00 INFO DockerTestBase [main] Removing docker network: 1cd3c07b468b7bf322657f22676ddf0c97997e5400fa40331eb409eaad36d89c 2021-06-09T14:33:10.627-07:00 INFO DockerTestBase [main] Removed docker network: 1cd3c07b468b7bf322657f22676ddf0c97997e5400fa40331eb409eaad36d89c 2021-06-09T14:33:10.634-07:00 WARN TestMethodTracer [main] ### upgradeVerifyClusterCompat SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 384.568 s - in com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFrom603ITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD 2021-06-09 14:33:14,414 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:33:14.499-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:33:14.535-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:33:14.536-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:33:14.727-07:00 WARN TestMethodTracer [main] ### failoverUpgradeAnalytics START 2021-06-09T14:33:14.763-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:33:16.071-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe) 2021-06-09T14:33:16.144-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe, id = 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 } 2021-06-09T14:33:16.152-07:00 INFO UpgradeTestBase [pool-0] Creating kv1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:33:16.161-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv1 to /opt/couchbase/var 2021-06-09T14:33:16.164-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull couchbase:enterprise-6.0.0 2021-06-09T14:33:16.165-07:00 INFO UpgradeTestBase [pool-2] Creating cbas2 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:33:16.166-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002cbas2 to /opt/couchbase/var 2021-06-09T14:33:16.160-07:00 INFO UpgradeTestBase [pool-1] Creating cbas1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:33:16.169-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003cbas1 to /opt/couchbase/var 2021-06-09T14:33:17.868-07:00 INFO DockerTestBase [pool-0+] >> enterprise-6.0.0: Pulling from library/couchbase 2021-06-09T14:33:18.234-07:00 INFO DockerTestBase [pool-0+] >> 7b722c1070cd: Pulling fs layer 2021-06-09T14:33:18.234-07:00 INFO DockerTestBase [pool-0+] >> 5fbf74db61f1: Pulling fs layer 2021-06-09T14:33:18.234-07:00 INFO DockerTestBase [pool-0+] >> ed41cb72e5c9: Pulling fs layer 2021-06-09T14:33:18.234-07:00 INFO DockerTestBase [pool-0+] >> 7ea47a67709e: Pulling fs layer 2021-06-09T14:33:18.235-07:00 INFO DockerTestBase [pool-0+] >> ca04de705515: Pulling fs layer 2021-06-09T14:33:18.235-07:00 INFO DockerTestBase [pool-0+] >> ce20af7f0ef5: Pulling fs layer 2021-06-09T14:33:18.235-07:00 INFO DockerTestBase [pool-0+] >> 397a78ce7d99: Pulling fs layer 2021-06-09T14:33:18.235-07:00 INFO DockerTestBase [pool-0+] >> 05a9f0b5672f: Pulling fs layer 2021-06-09T14:33:18.235-07:00 INFO DockerTestBase [pool-0+] >> 4c755e368998: Pulling fs layer 2021-06-09T14:33:18.235-07:00 INFO DockerTestBase [pool-0+] >> da66694ebab8: Pulling fs layer 2021-06-09T14:33:18.236-07:00 INFO DockerTestBase [pool-0+] >> 378756cb2ddd: Pulling fs layer 2021-06-09T14:33:18.236-07:00 INFO DockerTestBase [pool-0+] >> 819c17bd23b2: Pulling fs layer 2021-06-09T14:33:18.236-07:00 INFO DockerTestBase [pool-0+] >> 3c1b06acba80: Pulling fs layer 2021-06-09T14:33:18.236-07:00 INFO DockerTestBase [pool-0+] >> 7ea47a67709e: Waiting 2021-06-09T14:33:18.236-07:00 INFO DockerTestBase [pool-0+] >> 4c755e368998: Waiting 2021-06-09T14:33:18.236-07:00 INFO DockerTestBase [pool-0+] >> da66694ebab8: Waiting 2021-06-09T14:33:18.236-07:00 INFO DockerTestBase [pool-0+] >> ca04de705515: Waiting 2021-06-09T14:33:18.237-07:00 INFO DockerTestBase [pool-0+] >> ce20af7f0ef5: Waiting 2021-06-09T14:33:18.237-07:00 INFO DockerTestBase [pool-0+] >> 397a78ce7d99: Waiting 2021-06-09T14:33:18.237-07:00 INFO DockerTestBase [pool-0+] >> 378756cb2ddd: Waiting 2021-06-09T14:33:18.237-07:00 INFO DockerTestBase [pool-0+] >> 05a9f0b5672f: Waiting 2021-06-09T14:33:18.237-07:00 INFO DockerTestBase [pool-0+] >> 3c1b06acba80: Waiting 2021-06-09T14:33:18.237-07:00 INFO DockerTestBase [pool-0+] >> 819c17bd23b2: Waiting 2021-06-09T14:33:18.630-07:00 INFO DockerTestBase [pool-0+] >> 5fbf74db61f1: Verifying Checksum 2021-06-09T14:33:18.630-07:00 INFO DockerTestBase [pool-0+] >> 5fbf74db61f1: Download complete 2021-06-09T14:33:18.647-07:00 INFO DockerTestBase [pool-0+] >> ed41cb72e5c9: Download complete 2021-06-09T14:33:19.370-07:00 INFO DockerTestBase [pool-0+] >> 7b722c1070cd: Verifying Checksum 2021-06-09T14:33:19.372-07:00 INFO DockerTestBase [pool-0+] >> 7b722c1070cd: Download complete 2021-06-09T14:33:19.587-07:00 INFO DockerTestBase [pool-0+] >> ca04de705515: Verifying Checksum 2021-06-09T14:33:19.588-07:00 INFO DockerTestBase [pool-0+] >> ca04de705515: Download complete 2021-06-09T14:33:21.312-07:00 INFO DockerTestBase [pool-0+] >> 7b722c1070cd: Pull complete 2021-06-09T14:33:21.460-07:00 INFO DockerTestBase [pool-0+] >> 5fbf74db61f1: Pull complete 2021-06-09T14:33:21.880-07:00 INFO DockerTestBase [pool-0+] >> ed41cb72e5c9: Pull complete 2021-06-09T14:33:24.266-07:00 INFO DockerTestBase [pool-0+] >> ce20af7f0ef5: Verifying Checksum 2021-06-09T14:33:24.266-07:00 INFO DockerTestBase [pool-0+] >> ce20af7f0ef5: Download complete 2021-06-09T14:33:24.270-07:00 INFO DockerTestBase [pool-0+] >> 7ea47a67709e: Verifying Checksum 2021-06-09T14:33:24.270-07:00 INFO DockerTestBase [pool-0+] >> 7ea47a67709e: Download complete 2021-06-09T14:33:24.474-07:00 INFO DockerTestBase [pool-0+] >> 7ea47a67709e: Pull complete 2021-06-09T14:33:24.718-07:00 INFO DockerTestBase [pool-0+] >> 4c755e368998: Verifying Checksum 2021-06-09T14:33:24.718-07:00 INFO DockerTestBase [pool-0+] >> 4c755e368998: Download complete 2021-06-09T14:33:24.730-07:00 INFO DockerTestBase [pool-0+] >> 05a9f0b5672f: Verifying Checksum 2021-06-09T14:33:24.731-07:00 INFO DockerTestBase [pool-0+] >> 05a9f0b5672f: Download complete 2021-06-09T14:33:25.435-07:00 INFO DockerTestBase [pool-0+] >> 378756cb2ddd: Verifying Checksum 2021-06-09T14:33:25.436-07:00 INFO DockerTestBase [pool-0+] >> 378756cb2ddd: Download complete 2021-06-09T14:33:25.500-07:00 INFO DockerTestBase [pool-0+] >> da66694ebab8: Verifying Checksum 2021-06-09T14:33:25.501-07:00 INFO DockerTestBase [pool-0+] >> da66694ebab8: Download complete 2021-06-09T14:33:25.814-07:00 INFO DockerTestBase [pool-0+] >> ca04de705515: Pull complete 2021-06-09T14:33:25.877-07:00 INFO DockerTestBase [pool-0+] >> 819c17bd23b2: Verifying Checksum 2021-06-09T14:33:25.878-07:00 INFO DockerTestBase [pool-0+] >> 819c17bd23b2: Download complete 2021-06-09T14:33:25.924-07:00 INFO DockerTestBase [pool-0+] >> 3c1b06acba80: Verifying Checksum 2021-06-09T14:33:25.925-07:00 INFO DockerTestBase [pool-0+] >> 3c1b06acba80: Download complete 2021-06-09T14:33:26.006-07:00 INFO DockerTestBase [pool-0+] >> ce20af7f0ef5: Pull complete 2021-06-09T14:33:29.417-07:00 INFO DockerTestBase [pool-0+] >> 397a78ce7d99: Verifying Checksum 2021-06-09T14:33:29.418-07:00 INFO DockerTestBase [pool-0+] >> 397a78ce7d99: Download complete 2021-06-09T14:33:37.360-07:00 INFO DockerTestBase [pool-0+] >> 397a78ce7d99: Pull complete 2021-06-09T14:33:37.803-07:00 INFO DockerTestBase [pool-0+] >> 05a9f0b5672f: Pull complete 2021-06-09T14:33:38.012-07:00 INFO DockerTestBase [pool-0+] >> 4c755e368998: Pull complete 2021-06-09T14:33:38.277-07:00 INFO DockerTestBase [pool-0+] >> da66694ebab8: Pull complete 2021-06-09T14:33:38.440-07:00 INFO DockerTestBase [pool-0+] >> 378756cb2ddd: Pull complete 2021-06-09T14:33:38.660-07:00 INFO DockerTestBase [pool-0+] >> 819c17bd23b2: Pull complete 2021-06-09T14:33:38.803-07:00 INFO DockerTestBase [pool-0+] >> 3c1b06acba80: Pull complete 2021-06-09T14:33:38.810-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:342fb755176fb28c826b705721d69eb0fbba8631fd41f3a103806c235bbc6760 2021-06-09T14:33:38.814-07:00 INFO DockerTestBase [pool-0+] >> Status: Downloaded newer image for couchbase:enterprise-6.0.0 2021-06-09T14:33:38.815-07:00 INFO DockerTestBase [pool-0+] >> docker.io/library/couchbase:enterprise-6.0.0 2021-06-09T14:33:38.818-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0001_kv1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe, id = 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 }) 2021-06-09T14:33:38.819-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0003_cbas1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe, id = 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 }) 2021-06-09T14:33:38.819-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0002_cbas2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe, id = 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 }) 2021-06-09T14:33:38.846-07:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8091/tcp -> 172.17.0.1:42170/tcp 2021-06-09T14:33:38.846-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8091/tcp -> 172.17.0.1:33424/tcp 2021-06-09T14:33:38.846-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8091/tcp -> 172.17.0.1:42567/tcp 2021-06-09T14:33:38.846-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8095/tcp -> 172.17.0.1:45560/tcp 2021-06-09T14:33:38.846-07:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8095/tcp -> 172.17.0.1:40858/tcp 2021-06-09T14:33:38.847-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8095/tcp -> 172.17.0.1:33133/tcp 2021-06-09T14:33:39.032-07:00 INFO DockerTestBase [pool-2] Starting docker container: e500f06f49984 2021-06-09T14:33:39.036-07:00 INFO DockerTestBase [pool-1] Starting docker container: 3d2d5f7506777 2021-06-09T14:33:39.045-07:00 INFO DockerTestBase [pool-0] Starting docker container: c059fac98f3c3 2021-06-09T14:33:41.359-07:00 INFO DockerTestBase [pool-2] Started docker container: e500f06f49984 2021-06-09T14:33:41.458-07:00 INFO DockerTestBase [pool-0] Started docker container: c059fac98f3c3 2021-06-09T14:33:41.471-07:00 INFO DockerTestBase [pool-1] Started docker container: 3d2d5f7506777 2021-06-09T14:33:41.628-07:00 INFO DockerTestBase [pool-2] Container information for e500f06f49984: host: cbas2.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:33133->192.168.176.2:8095, 172.17.0.1:33424->192.168.176.2:8091 2021-06-09T14:33:41.629-07:00 INFO DockerTestBase [pool-0] Container information for c059fac98f3c3: host: kv1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.4, 172.17.0.1:45560->192.168.176.4:8095, 172.17.0.1:42567->192.168.176.4:8091 2021-06-09T14:33:41.647-07:00 INFO DockerTestBase [pool-1] Container information for 3d2d5f7506777: host: cbas1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:40858->192.168.176.3:8095, 172.17.0.1:42170->192.168.176.3:8091 2021-06-09T14:33:54.355-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node cbas1 (192.168.176.3) 2021-06-09T14:33:54.365-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cbas1.couchbase.host 2021-06-09T14:33:55.002-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node cbas2 (192.168.176.2) 2021-06-09T14:33:55.002-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec cbas2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname cbas2.couchbase.host 2021-06-09T14:33:55.483-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T14:33:55.529-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node kv1 (192.168.176.4) 2021-06-09T14:33:55.529-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname kv1.couchbase.host 2021-06-09T14:33:55.967-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T14:33:56.405-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:33:56.417-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node kv1 (192.168.176.4) with services: [DATA] 2021-06-09T14:33:56.418-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:33:56.856-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T14:33:56.870-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node cbas1 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:33:56.870-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://cbas1.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:33:56.871-07:00 INFO DockerTestBase [pool-2] Running 'server-add' for node cbas2 (192.168.176.2) with services: [ANALYTICS] 2021-06-09T14:33:56.871-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://cbas2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:34:00.762-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Server added 2021-06-09T14:34:04.399-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T14:34:04.409-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:34:04.410-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase 2021-06-09T14:34:05.874-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:34:05.874-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:34:05.908-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:34:05.948-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:34:05.949-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:34:05.977-07:00 INFO DockerTestBase [pool-5] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:34:06.051-07:00 INFO KvStoreHttpHelper [pool-2] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42170] 2021-06-09T14:34:06.065-07:00 INFO KvStoreHttpHelper [pool-2] installing sample(s): [beer-sample] on 172.17.0.1:42170 2021-06-09T14:34:15.423-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42170] 2021-06-09T14:34:18.766-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:34:19.228-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:34:19.233-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:34:20.724-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1122 } 2021-06-09T14:34:20.724-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 1122 } 2021-06-09T14:34:21.871-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:34:21.871-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:34:21.872-07:00 INFO UpgradeFailoverTestBase [main] Failing over (hard) cbas nodes ... 2021-06-09T14:34:21.887-07:00 INFO DockerTestBase [main] Running 'failover --force' on [cbas1 (192.168.176.3), cbas2 (192.168.176.2)] nodes 2021-06-09T14:34:21.889-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli failover -c 192.168.176.4:8091 -u couchbase -p couchbase --server-failover cbas1.couchbase.host,cbas2.couchbase.host --force 2021-06-09T14:34:22.446-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server failed over 2021-06-09T14:34:27.460-07:00 INFO UpgradeTestBase [main] Recreating 'cbas2' node (image = null, forceNewIp = false)... 2021-06-09T14:34:27.460-07:00 INFO DockerTestBase [main] Copying files from docker container e500f06f49984: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0002_cbas2_couchbase_enterprise-6.0.0 2021-06-09T14:34:27.464-07:00 INFO UpgradeTestBase [ForkJoinPool.commonPool-worker-3] Recreating 'cbas1' node (image = null, forceNewIp = false)... 2021-06-09T14:34:27.465-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 3d2d5f7506777: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0003_cbas1_couchbase_enterprise-6.0.0 2021-06-09T14:34:27.780-07:00 INFO DockerTestBase [main] Stopping docker container: e500f06f49984 2021-06-09T14:34:27.821-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 3d2d5f7506777 2021-06-09T14:34:28.273-07:00 INFO DockerTestBase [main] Stopped docker container: e500f06f49984 2021-06-09T14:34:28.274-07:00 INFO DockerTestBase [main] Removing docker container: e500f06f49984 2021-06-09T14:34:28.298-07:00 INFO DockerTestBase [main] Removed docker container: e500f06f49984 2021-06-09T14:34:28.302-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:34:28.366-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 3d2d5f7506777 2021-06-09T14:34:28.366-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 3d2d5f7506777 2021-06-09T14:34:28.379-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 3d2d5f7506777 2021-06-09T14:34:28.419-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:34:28.421-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:34:28.421-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:34:28.423-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:34:28.426-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0004_cbas2_thisbuild, network = { name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe, id = 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 }) 2021-06-09T14:34:28.426-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0005_cbas1_thisbuild, network = { name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe, id = 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 }) 2021-06-09T14:34:28.427-07:00 INFO DockerTestBase [main] binding exposed port cbas2:8091/tcp -> 172.17.0.1:43304/tcp 2021-06-09T14:34:28.427-07:00 INFO DockerTestBase [main] binding exposed port cbas2:8095/tcp -> 172.17.0.1:37366/tcp 2021-06-09T14:34:28.427-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8091/tcp -> 172.17.0.1:42396/tcp 2021-06-09T14:34:28.428-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8095/tcp -> 172.17.0.1:35640/tcp 2021-06-09T14:34:28.467-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Starting docker container: d52d56f4eb067 2021-06-09T14:34:28.473-07:00 INFO DockerTestBase [main] Starting docker container: 8fb3f110cb03e 2021-06-09T14:34:30.223-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Started docker container: d52d56f4eb067 2021-06-09T14:34:30.246-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Container information for d52d56f4eb067: host: cbas1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:35640->192.168.176.2:8095, 172.17.0.1:42396->192.168.176.2:8091 2021-06-09T14:34:31.032-07:00 INFO DockerTestBase [main] Started docker container: 8fb3f110cb03e 2021-06-09T14:34:31.044-07:00 INFO DockerTestBase [main] Container information for 8fb3f110cb03e: host: cbas2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:37366->192.168.176.3:8095, 172.17.0.1:43304->192.168.176.3:8091 2021-06-09T14:34:43.104-07:00 INFO DockerTestBase [main] Running full 'recovery' on nodes [cbas1 (192.168.176.3), cbas2 (192.168.176.2)] 2021-06-09T14:34:43.105-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli recovery -c 192.168.176.4:8091 -u couchbase -p couchbase --server-recovery cbas1.couchbase.host,cbas2.couchbase.host --recovery-type full 2021-06-09T14:34:43.635-07:00 INFO DockerTestBase [main+] >> SUCCESS: Servers recovered 2021-06-09T14:34:43.644-07:00 INFO UpgradeTestBase [main] Upgrading [kv1.couchbase.host] (online, rebalance) ... 2021-06-09T14:34:43.646-07:00 INFO UpgradeTestBase [pool-5] Creating kv1-new node using 'this build' docker image... 2021-06-09T14:34:43.646-07:00 INFO UpgradeTestBase [pool-5] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004kv1new to /opt/couchbase/var 2021-06-09T14:34:43.647-07:00 INFO DockerTestBase [pool-5] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0006_kv1new_thisbuild, network = { name = cbas-docker-test-0001-2af0a037-28b6-4967-ad5a-eaebdc6ccdfe, id = 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 }) 2021-06-09T14:34:43.647-07:00 INFO DockerTestBase [pool-5] binding exposed port kv1new:8091/tcp -> 172.17.0.1:44691/tcp 2021-06-09T14:34:43.648-07:00 INFO DockerTestBase [pool-5] binding exposed port kv1new:8095/tcp -> 172.17.0.1:39598/tcp 2021-06-09T14:34:43.699-07:00 INFO DockerTestBase [pool-5] Starting docker container: 62c2d6beabb32 2021-06-09T14:34:44.780-07:00 INFO DockerTestBase [pool-5] Started docker container: 62c2d6beabb32 2021-06-09T14:34:44.789-07:00 INFO DockerTestBase [pool-5] Container information for 62c2d6beabb32: host: kv1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.5, 172.17.0.1:39598->192.168.176.5:8095, 172.17.0.1:44691->192.168.176.5:8091 2021-06-09T14:34:54.849-07:00 INFO DockerTestBase [pool-5] Running 'node-init' on node kv1new (192.168.176.5) 2021-06-09T14:34:54.850-07:00 INFO DockerTestBase [pool-5] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname kv1new.couchbase.host 2021-06-09T14:34:55.628-07:00 INFO DockerTestBase [pool-5+] >> SUCCESS: Node initialized 2021-06-09T14:34:55.674-07:00 INFO DockerTestBase [pool-5] Running 'server-add' for node kv1new (192.168.176.5) with services: [DATA] 2021-06-09T14:34:55.674-07:00 INFO DockerTestBase [pool-5] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://kv1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T14:35:00.475-07:00 INFO DockerTestBase [pool-5+] >> SUCCESS: Server added 2021-06-09T14:35:00.508-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T14:35:00.508-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:35:00.510-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase --server-remove kv1.couchbase.host 2021-06-09T14:36:36.594-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:36:36.594-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:36:36.664-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:36:36.689-07:00 INFO DockerTestBase [pool-6] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:36:36.690-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:36:36.691-07:00 INFO DockerTestBase [main] Copying files from docker container c059fac98f3c3: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0001_kv1_couchbase_enterprise-6.0.0 2021-06-09T14:36:37.319-07:00 INFO DockerTestBase [main] Stopping docker container: c059fac98f3c3 2021-06-09T14:36:37.729-07:00 INFO DockerTestBase [main] Stopped docker container: c059fac98f3c3 2021-06-09T14:36:37.729-07:00 INFO DockerTestBase [main] Removing docker container: c059fac98f3c3 2021-06-09T14:36:37.748-07:00 INFO DockerTestBase [main] Removed docker container: c059fac98f3c3 2021-06-09T14:36:37.755-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T14:36:37.764-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T14:36:37.787-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:36:37.966-07:00 INFO DockerTestBase [pool-6] assuming 6.0 compatibility on missing 'clusterCompatibility' field 2021-06-09T14:36:38.793-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:36:39.142-07:00 INFO DockerTestBase [pool-6] assuming 6.0 compatibility on missing 'clusterCompatibility' field 2021-06-09T14:36:39.796-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:40.328-07:00 INFO DockerTestBase [pool-6] assuming 6.0 compatibility on missing 'clusterCompatibility' field 2021-06-09T14:36:40.801-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:41.398-07:00 INFO DockerTestBase [pool-6] assuming 6.0 compatibility on missing 'clusterCompatibility' field 2021-06-09T14:36:41.806-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:42.413-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ed26318[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@2015996a] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:42.826-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:36:43.428-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@28118a00[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@26d670aa] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:43.830-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:44.446-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@49556260[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@479294e2] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:44.834-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:45.469-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@73686d9b[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@7daf8c5e] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:45.838-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:46.485-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@c1b2642[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@2dcc869] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:46.842-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:47.498-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@417a013f[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@4fbdf6b8] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:47.845-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:48.512-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4fb440a9[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@4173b6f7] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:48.849-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:49.528-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@251dedde[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@2bda06d5] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:49.853-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:50.541-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e060d80[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@c1f3e8] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:50.857-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:51.553-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@715661b0[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$514/0x0000000100909c40@787c7543] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@536d99eb[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 27] 2021-06-09T14:36:51.861-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:52.558-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:36:52.864-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:53.562-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:53.868-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:54.566-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:36:54.872-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:55.570-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:55.876-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:56.574-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:56.883-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:57.581-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:57.887-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:58.585-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:36:58.890-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:36:59.590-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:36:59.894-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:00.593-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:00.898-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:01.600-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:01.902-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:02.604-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:02.905-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:03.922-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:04.926-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:05.930-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:06.933-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:07.937-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:08.942-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:09.947-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:10.951-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:11.956-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:12.960-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:13.964-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:14.968-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:15.971-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:16.974-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:17.978-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:18.982-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:19.985-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:20.989-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:21.995-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:22.998-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:24.003-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:25.006-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:26.023-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:27.028-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:28.032-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:29.036-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:30.039-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:31.043-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:32.047-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:33.051-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:34.055-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:35.059-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:36.063-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:37.067-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:38.071-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:39.075-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:40.079-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:41.083-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:42.087-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:43.090-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:44.094-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:45.098-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:46.101-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:47.105-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:48.108-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:49.112-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:50.115-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:51.118-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:52.121-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:53.124-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:37:54.128-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:37:55.456-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:37:55.456-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:37:56.836-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42396] 2021-06-09T14:37:56.837-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T14:37:56.837-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:43304] 2021-06-09T14:37:56.837-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:37:57.669-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 0 } 2021-06-09T14:37:57.670-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:37:57.692-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:37:57.692-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:37:58.980-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 0 } 2021-06-09T14:37:58.981-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 2. ResultSet found : { "$1": 0 } 2021-06-09T14:37:59.118-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:37:59.118-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 0 } 2021-06-09T14:38:00.338-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 418 } 2021-06-09T14:38:00.338-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 3. ResultSet found : { "$1": 418 } 2021-06-09T14:38:00.440-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 605 } 2021-06-09T14:38:00.440-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 605 } 2021-06-09T14:38:01.567-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 3259 } 2021-06-09T14:38:01.568-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 4. ResultSet found : { "$1": 3259 } 2021-06-09T14:38:01.658-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3501 } 2021-06-09T14:38:01.658-07:00 INFO DockerTestBase [main] Failed attempt number: 4. ResultSet found : { "$1": 3501 } 2021-06-09T14:38:02.714-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T14:38:02.714-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T14:38:02.831-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:38:02.832-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:38:02.834-07:00 INFO DockerTestBase [main] Copying files from docker container 8fb3f110cb03e: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0004_cbas2_thisbuild 2021-06-09T14:38:02.835-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container d52d56f4eb067: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0005_cbas1_thisbuild 2021-06-09T14:38:02.834-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container 62c2d6beabb32: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAnalytics/0006_kv1new_thisbuild 2021-06-09T14:38:03.419-07:00 INFO DockerTestBase [main] Stopping docker container: 8fb3f110cb03e 2021-06-09T14:38:03.909-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: 62c2d6beabb32 2021-06-09T14:38:04.090-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: d52d56f4eb067 2021-06-09T14:38:04.192-07:00 INFO DockerTestBase [main] Stopped docker container: 8fb3f110cb03e 2021-06-09T14:38:04.495-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: 62c2d6beabb32 2021-06-09T14:38:04.597-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: d52d56f4eb067 2021-06-09T14:38:04.598-07:00 INFO DockerTestBase [main] Removing docker container: 8fb3f110cb03e 2021-06-09T14:38:04.598-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: d52d56f4eb067 2021-06-09T14:38:04.601-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 62c2d6beabb32 2021-06-09T14:38:04.612-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: d52d56f4eb067 2021-06-09T14:38:04.613-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 62c2d6beabb32 2021-06-09T14:38:04.613-07:00 INFO DockerTestBase [main] Removed docker container: 8fb3f110cb03e 2021-06-09T14:38:04.615-07:00 INFO DockerTestBase [main] Removing docker network: 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 2021-06-09T14:38:04.698-07:00 INFO DockerTestBase [main] Removed docker network: 7bb6d95eaf4267eaa24c2fa95a0207a0325ad3a97051b7eea80df238360dfc26 2021-06-09T14:38:04.704-07:00 WARN TestMethodTracer [main] ### failoverUpgradeAnalytics SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 292.909 s - in com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD 2021-06-09 14:38:07,652 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:38:07.732-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:38:07.768-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:38:07.768-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:38:07.932-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.0.0 2021-06-09T14:38:09.046-07:00 INFO DockerTestBase [main+] >> enterprise-6.0.0: Pulling from library/couchbase 2021-06-09T14:38:09.049-07:00 INFO DockerTestBase [main+] >> Digest: sha256:342fb755176fb28c826b705721d69eb0fbba8631fd41f3a103806c235bbc6760 2021-06-09T14:38:09.049-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for couchbase:enterprise-6.0.0 2021-06-09T14:38:09.051-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.0.0 2021-06-09T14:38:09.059-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutSingletonCBASOnlyOffline START 2021-06-09T14:38:09.111-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:38:10.300-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-4968f773-19a5-4c16-9948-9bdbafb0d9d8) 2021-06-09T14:38:10.377-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-4968f773-19a5-4c16-9948-9bdbafb0d9d8, id = e9e999aa73d5dddbb22346be028ba283979fe1b2c2aeb8a5c5a51738860fb533 } 2021-06-09T14:38:10.377-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #1 ... 2021-06-09T14:38:10.378-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:38:10.380-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:38:10.383-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutSingletonCBASOnlyOffline/0001_n1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-4968f773-19a5-4c16-9948-9bdbafb0d9d8, id = e9e999aa73d5dddbb22346be028ba283979fe1b2c2aeb8a5c5a51738860fb533 }) 2021-06-09T14:38:10.403-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:41764/tcp 2021-06-09T14:38:10.404-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:40263/tcp 2021-06-09T14:38:10.538-07:00 INFO DockerTestBase [main] Starting docker container: e5578304d277f 2021-06-09T14:38:11.541-07:00 INFO DockerTestBase [main] Started docker container: e5578304d277f 2021-06-09T14:38:11.665-07:00 INFO DockerTestBase [main] Container information for e5578304d277f: host: n1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:40263->192.168.176.2:8095, 172.17.0.1:41764->192.168.176.2:8091 2021-06-09T14:38:22.803-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:38:22.804-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:38:23.399-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:38:23.409-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:38:23.410-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:38:24.451-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:38:24.460-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #2 ... 2021-06-09T14:38:24.461-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:38:24.461-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:38:24.462-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutSingletonCBASOnlyOffline/0002_n2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-4968f773-19a5-4c16-9948-9bdbafb0d9d8, id = e9e999aa73d5dddbb22346be028ba283979fe1b2c2aeb8a5c5a51738860fb533 }) 2021-06-09T14:38:24.463-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:40740/tcp 2021-06-09T14:38:24.463-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:33265/tcp 2021-06-09T14:38:24.532-07:00 INFO DockerTestBase [main] Starting docker container: de721c2ea214d 2021-06-09T14:38:25.527-07:00 INFO DockerTestBase [main] Started docker container: de721c2ea214d 2021-06-09T14:38:25.542-07:00 INFO DockerTestBase [main] Container information for de721c2ea214d: host: n2.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:33265->192.168.176.3:8095, 172.17.0.1:40740->192.168.176.3:8091 2021-06-09T14:38:36.836-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:38:36.837-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:38:37.875-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:38:37.884-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:38:37.885-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:38:41.104-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:38:41.112-07:00 INFO UpgradeFromAlice600ITD [main] Rebalancing ... 2021-06-09T14:38:41.113-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:38:41.113-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:38:42.454-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:38:42.455-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:38:42.481-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:38:42.505-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:38:42.506-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:38:42.515-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:38:49.757-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:41764] 2021-06-09T14:38:49.761-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:41764 2021-06-09T14:38:49.837-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:40740] 2021-06-09T14:38:53.680-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:38:54.055-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:38:54.060-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:38:55.348-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3560 } 2021-06-09T14:38:55.349-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 3560 } 2021-06-09T14:38:56.473-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:38:56.474-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:38:56.475-07:00 INFO UpgradeTestBase [main] Recreating 'n2' node (image = null, forceNewIp = false)... 2021-06-09T14:38:56.476-07:00 INFO DockerTestBase [main] Copying files from docker container de721c2ea214d: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutSingletonCBASOnlyOffline/0002_n2_couchbase_enterprise-6.0.0 2021-06-09T14:38:56.701-07:00 INFO DockerTestBase [main] Stopping docker container: de721c2ea214d 2021-06-09T14:38:57.109-07:00 INFO DockerTestBase [main] Stopped docker container: de721c2ea214d 2021-06-09T14:38:57.109-07:00 INFO DockerTestBase [main] Removing docker container: de721c2ea214d 2021-06-09T14:38:57.127-07:00 INFO DockerTestBase [main] Removed docker container: de721c2ea214d 2021-06-09T14:38:57.129-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:38:57.237-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:38:57.238-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:38:57.238-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:38:57.238-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:38:57.246-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutSingletonCBASOnlyOffline/0003_n2_thisbuild, network = { name = cbas-docker-test-0001-4968f773-19a5-4c16-9948-9bdbafb0d9d8, id = e9e999aa73d5dddbb22346be028ba283979fe1b2c2aeb8a5c5a51738860fb533 }) 2021-06-09T14:38:57.247-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:46056/tcp 2021-06-09T14:38:57.247-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:39157/tcp 2021-06-09T14:38:57.308-07:00 INFO DockerTestBase [main] Starting docker container: 752f4e0015318 2021-06-09T14:38:58.433-07:00 INFO DockerTestBase [main] Started docker container: 752f4e0015318 2021-06-09T14:38:58.445-07:00 INFO DockerTestBase [main] Container information for 752f4e0015318: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:39157->192.168.176.3:8095, 172.17.0.1:46056->192.168.176.3:8091 2021-06-09T14:39:08.487-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:39:14.884-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46056] 2021-06-09T14:39:14.885-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:39:15.813-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:39:15.813-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:39:17.042-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:39:17.042-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 0 } 2021-06-09T14:39:18.293-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1692 } 2021-06-09T14:39:18.293-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 1692 } 2021-06-09T14:39:19.385-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:39:19.385-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:39:19.395-07:00 INFO DockerTestBase [main] Copying files from docker container 752f4e0015318: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutSingletonCBASOnlyOffline/0003_n2_thisbuild 2021-06-09T14:39:19.396-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container e5578304d277f: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutSingletonCBASOnlyOffline/0001_n1_couchbase_enterprise-6.0.0 2021-06-09T14:39:19.683-07:00 INFO DockerTestBase [main] Stopping docker container: 752f4e0015318 2021-06-09T14:39:19.706-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: e5578304d277f 2021-06-09T14:39:20.311-07:00 INFO DockerTestBase [main] Stopped docker container: 752f4e0015318 2021-06-09T14:39:20.359-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: e5578304d277f 2021-06-09T14:39:20.361-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: e5578304d277f 2021-06-09T14:39:20.361-07:00 INFO DockerTestBase [main] Removing docker container: 752f4e0015318 2021-06-09T14:39:20.371-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: e5578304d277f 2021-06-09T14:39:20.372-07:00 INFO DockerTestBase [main] Removed docker container: 752f4e0015318 2021-06-09T14:39:20.374-07:00 INFO DockerTestBase [main] Removing docker network: e9e999aa73d5dddbb22346be028ba283979fe1b2c2aeb8a5c5a51738860fb533 2021-06-09T14:39:20.449-07:00 INFO DockerTestBase [main] Removed docker network: e9e999aa73d5dddbb22346be028ba283979fe1b2c2aeb8a5c5a51738860fb533 2021-06-09T14:39:20.457-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutSingletonCBASOnlyOffline SUCCEEDED 2021-06-09T14:39:20.458-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyOffline START 2021-06-09T14:39:20.460-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:39:20.514-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0002-e1bfe5e7-0ad8-4656-bf77-b299fa3cf3e6) 2021-06-09T14:39:20.565-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0002-e1bfe5e7-0ad8-4656-bf77-b299fa3cf3e6, id = 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef } 2021-06-09T14:39:20.566-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #1 ... 2021-06-09T14:39:20.566-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:39:20.623-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:39:20.624-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0004_n1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0002-e1bfe5e7-0ad8-4656-bf77-b299fa3cf3e6, id = 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef }) 2021-06-09T14:39:20.628-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:43709/tcp 2021-06-09T14:39:20.629-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:36246/tcp 2021-06-09T14:39:20.702-07:00 INFO DockerTestBase [main] Starting docker container: 2b7dd475d58ad 2021-06-09T14:39:21.745-07:00 INFO DockerTestBase [main] Started docker container: 2b7dd475d58ad 2021-06-09T14:39:21.793-07:00 INFO DockerTestBase [main] Container information for 2b7dd475d58ad: host: n1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:36246->192.168.176.2:8095, 172.17.0.1:43709->192.168.176.2:8091 2021-06-09T14:39:32.834-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:39:32.834-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:39:33.489-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:39:33.496-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:39:33.496-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:39:34.546-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:39:34.557-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #2 ... 2021-06-09T14:39:34.557-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:39:34.571-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:39:34.572-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0005_n2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0002-e1bfe5e7-0ad8-4656-bf77-b299fa3cf3e6, id = 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef }) 2021-06-09T14:39:34.573-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:34298/tcp 2021-06-09T14:39:34.573-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:35252/tcp 2021-06-09T14:39:34.637-07:00 INFO DockerTestBase [main] Starting docker container: da7c6878918a1 2021-06-09T14:39:35.714-07:00 INFO DockerTestBase [main] Started docker container: da7c6878918a1 2021-06-09T14:39:35.720-07:00 INFO DockerTestBase [main] Container information for da7c6878918a1: host: n2.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:35252->192.168.176.3:8095, 172.17.0.1:34298->192.168.176.3:8091 2021-06-09T14:39:46.765-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:39:46.766-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:39:47.519-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:39:47.528-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #3 ... 2021-06-09T14:39:47.529-07:00 INFO UpgradeTestBase [main] Creating n3 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:39:47.529-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n3 to /opt/couchbase/var 2021-06-09T14:39:47.530-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0006_n3_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0002-e1bfe5e7-0ad8-4656-bf77-b299fa3cf3e6, id = 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef }) 2021-06-09T14:39:47.530-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:40434/tcp 2021-06-09T14:39:47.531-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:41518/tcp 2021-06-09T14:39:47.600-07:00 INFO DockerTestBase [main] Starting docker container: 35eb7c65e97f9 2021-06-09T14:39:48.386-07:00 INFO DockerTestBase [main] Started docker container: 35eb7c65e97f9 2021-06-09T14:39:48.400-07:00 INFO DockerTestBase [main] Container information for 35eb7c65e97f9: host: n3.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.4, 172.17.0.1:41518->192.168.176.4:8095, 172.17.0.1:40434->192.168.176.4:8091 2021-06-09T14:39:58.469-07:00 INFO DockerTestBase [main] Running 'node-init' on node n3 (192.168.176.4) 2021-06-09T14:39:58.470-07:00 INFO DockerTestBase [main] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n3.couchbase.host 2021-06-09T14:39:59.002-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:39:59.014-07:00 INFO UpgradeFromAlice600ITD [main] Running server-add ... 2021-06-09T14:39:59.014-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:39:59.014-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:40:01.993-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:40:02.001-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:40:02.002-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n3.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:40:05.291-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:40:05.302-07:00 INFO UpgradeFromAlice600ITD [main] Rebalancing ... 2021-06-09T14:40:05.302-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:40:05.302-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:40:06.674-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:40:06.674-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:40:06.684-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:40:06.691-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:40:06.692-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:40:06.693-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:40:13.292-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:43709] 2021-06-09T14:40:13.293-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:43709 2021-06-09T14:40:13.354-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:34298] 2021-06-09T14:40:17.124-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:40:17.733-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:40:17.734-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:40:19.031-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 818 } 2021-06-09T14:40:19.032-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 818 } 2021-06-09T14:40:20.231-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5479 } 2021-06-09T14:40:20.232-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 5479 } 2021-06-09T14:40:21.395-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:40:21.395-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:40:21.396-07:00 INFO UpgradeFromAlice600ITD [main] Upgrading CBAS nodes ... 2021-06-09T14:40:21.396-07:00 INFO UpgradeTestBase [main] Recreating 'n2' node (image = null, forceNewIp = false)... 2021-06-09T14:40:21.396-07:00 INFO DockerTestBase [main] Copying files from docker container da7c6878918a1: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0005_n2_couchbase_enterprise-6.0.0 2021-06-09T14:40:21.600-07:00 INFO DockerTestBase [main] Stopping docker container: da7c6878918a1 2021-06-09T14:40:22.026-07:00 INFO DockerTestBase [main] Stopped docker container: da7c6878918a1 2021-06-09T14:40:22.026-07:00 INFO DockerTestBase [main] Removing docker container: da7c6878918a1 2021-06-09T14:40:22.036-07:00 INFO DockerTestBase [main] Removed docker container: da7c6878918a1 2021-06-09T14:40:22.037-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0007_n2_thisbuild, network = { name = cbas-docker-test-0002-e1bfe5e7-0ad8-4656-bf77-b299fa3cf3e6, id = 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef }) 2021-06-09T14:40:22.038-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:35067/tcp 2021-06-09T14:40:22.039-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:35540/tcp 2021-06-09T14:40:22.092-07:00 INFO DockerTestBase [main] Starting docker container: 4ec6d9e08e798 2021-06-09T14:40:23.154-07:00 INFO DockerTestBase [main] Started docker container: 4ec6d9e08e798 2021-06-09T14:40:23.170-07:00 INFO DockerTestBase [main] Container information for 4ec6d9e08e798: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:35540->192.168.176.3:8095, 172.17.0.1:35067->192.168.176.3:8091 2021-06-09T14:40:33.451-07:00 INFO UpgradeTestBase [main] Recreating 'n3' node (image = null, forceNewIp = false)... 2021-06-09T14:40:33.452-07:00 INFO DockerTestBase [main] Copying files from docker container 35eb7c65e97f9: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0006_n3_couchbase_enterprise-6.0.0 2021-06-09T14:40:33.716-07:00 INFO DockerTestBase [main] Stopping docker container: 35eb7c65e97f9 2021-06-09T14:40:34.204-07:00 INFO DockerTestBase [main] Stopped docker container: 35eb7c65e97f9 2021-06-09T14:40:34.204-07:00 INFO DockerTestBase [main] Removing docker container: 35eb7c65e97f9 2021-06-09T14:40:34.217-07:00 INFO DockerTestBase [main] Removed docker container: 35eb7c65e97f9 2021-06-09T14:40:34.220-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0008_n3_thisbuild, network = { name = cbas-docker-test-0002-e1bfe5e7-0ad8-4656-bf77-b299fa3cf3e6, id = 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef }) 2021-06-09T14:40:34.221-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:39605/tcp 2021-06-09T14:40:34.221-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:42985/tcp 2021-06-09T14:40:34.268-07:00 INFO DockerTestBase [main] Starting docker container: 32f5afb872f19 2021-06-09T14:40:35.275-07:00 INFO DockerTestBase [main] Started docker container: 32f5afb872f19 2021-06-09T14:40:35.286-07:00 INFO DockerTestBase [main] Container information for 32f5afb872f19: host: n3.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:42985->192.168.176.4:8095, 172.17.0.1:39605->192.168.176.4:8091 2021-06-09T14:40:45.327-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:40:54.818-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35067] 2021-06-09T14:40:54.818-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:40:55.546-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2942 } 2021-06-09T14:40:55.546-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 2942 } 2021-06-09T14:40:56.865-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2942 } 2021-06-09T14:40:56.866-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2942 } 2021-06-09T14:40:58.272-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3797 } 2021-06-09T14:40:58.273-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 3797 } 2021-06-09T14:40:59.565-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5703 } 2021-06-09T14:40:59.566-07:00 INFO DockerTestBase [main] Failed attempt number: 4. ResultSet found : { "$1": 5703 } 2021-06-09T14:41:00.695-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:41:00.695-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:41:00.696-07:00 INFO DockerTestBase [main] Copying files from docker container 4ec6d9e08e798: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0007_n2_thisbuild 2021-06-09T14:41:00.703-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Copying files from docker container 32f5afb872f19: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0008_n3_thisbuild 2021-06-09T14:41:00.705-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 2b7dd475d58ad: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyOffline/0004_n1_couchbase_enterprise-6.0.0 2021-06-09T14:41:00.969-07:00 INFO DockerTestBase [main] Stopping docker container: 4ec6d9e08e798 2021-06-09T14:41:00.997-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 2b7dd475d58ad 2021-06-09T14:41:01.053-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopping docker container: 32f5afb872f19 2021-06-09T14:41:01.683-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 2b7dd475d58ad 2021-06-09T14:41:01.736-07:00 INFO DockerTestBase [main] Stopped docker container: 4ec6d9e08e798 2021-06-09T14:41:01.759-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopped docker container: 32f5afb872f19 2021-06-09T14:41:01.760-07:00 INFO DockerTestBase [main] Removing docker container: 4ec6d9e08e798 2021-06-09T14:41:01.760-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 32f5afb872f19 2021-06-09T14:41:01.763-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 2b7dd475d58ad 2021-06-09T14:41:01.776-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 32f5afb872f19 2021-06-09T14:41:01.776-07:00 INFO DockerTestBase [main] Removed docker container: 4ec6d9e08e798 2021-06-09T14:41:01.778-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 2b7dd475d58ad 2021-06-09T14:41:01.779-07:00 INFO DockerTestBase [main] Removing docker network: 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef 2021-06-09T14:41:01.850-07:00 INFO DockerTestBase [main] Removed docker network: 0a6f4948713bc1cdceb95e88c0e37797a1c34deec925fad23f83b5e30f8863ef 2021-06-09T14:41:01.852-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyOffline SUCCEEDED 2021-06-09T14:41:01.853-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyCCNewMixedOffline START 2021-06-09T14:41:01.855-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:41:01.899-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0003-965913b1-4b13-46ea-97dd-9b425ea466fb) 2021-06-09T14:41:01.948-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0003-965913b1-4b13-46ea-97dd-9b425ea466fb, id = 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 } 2021-06-09T14:41:01.949-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #1 ... 2021-06-09T14:41:01.949-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:41:02.001-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:41:02.002-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0009_n1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-965913b1-4b13-46ea-97dd-9b425ea466fb, id = 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 }) 2021-06-09T14:41:02.005-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:37060/tcp 2021-06-09T14:41:02.006-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:44909/tcp 2021-06-09T14:41:02.070-07:00 INFO DockerTestBase [main] Starting docker container: 6c14fea7b2329 2021-06-09T14:41:03.073-07:00 INFO DockerTestBase [main] Started docker container: 6c14fea7b2329 2021-06-09T14:41:03.109-07:00 INFO DockerTestBase [main] Container information for 6c14fea7b2329: host: n1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:44909->192.168.176.2:8095, 172.17.0.1:37060->192.168.176.2:8091 2021-06-09T14:41:14.147-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:41:14.147-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:41:14.783-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:41:14.791-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:41:14.792-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:41:15.834-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:41:15.842-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #2 ... 2021-06-09T14:41:15.842-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:41:15.861-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:41:15.862-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0010_n2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-965913b1-4b13-46ea-97dd-9b425ea466fb, id = 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 }) 2021-06-09T14:41:15.863-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:36628/tcp 2021-06-09T14:41:15.863-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:35314/tcp 2021-06-09T14:41:15.915-07:00 INFO DockerTestBase [main] Starting docker container: a8e4113e6d5f7 2021-06-09T14:41:17.004-07:00 INFO DockerTestBase [main] Started docker container: a8e4113e6d5f7 2021-06-09T14:41:17.014-07:00 INFO DockerTestBase [main] Container information for a8e4113e6d5f7: host: n2.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:35314->192.168.176.3:8095, 172.17.0.1:36628->192.168.176.3:8091 2021-06-09T14:41:28.054-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:41:28.054-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:41:28.741-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:41:28.750-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:41:28.750-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:41:31.628-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:41:31.637-07:00 INFO UpgradeFromAlice600ITD [main] Rebalancing ... 2021-06-09T14:41:31.637-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:41:31.638-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:41:32.979-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:41:32.979-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:41:32.986-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:41:32.991-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:41:32.993-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:41:32.993-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #3 ... 2021-06-09T14:41:32.993-07:00 INFO UpgradeTestBase [main] Creating n3 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:41:33.010-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n3 to /opt/couchbase/var 2021-06-09T14:41:33.011-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0011_n3_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-965913b1-4b13-46ea-97dd-9b425ea466fb, id = 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 }) 2021-06-09T14:41:33.012-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:39793/tcp 2021-06-09T14:41:33.012-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:33044/tcp 2021-06-09T14:41:33.067-07:00 INFO DockerTestBase [main] Starting docker container: e0be3918e2944 2021-06-09T14:41:33.707-07:00 INFO DockerTestBase [main] Started docker container: e0be3918e2944 2021-06-09T14:41:33.713-07:00 INFO DockerTestBase [main] Container information for e0be3918e2944: host: n3.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.4, 172.17.0.1:33044->192.168.176.4:8095, 172.17.0.1:39793->192.168.176.4:8091 2021-06-09T14:41:43.750-07:00 INFO DockerTestBase [main] Running 'node-init' on node n3 (192.168.176.4) 2021-06-09T14:41:43.750-07:00 INFO DockerTestBase [main] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n3.couchbase.host 2021-06-09T14:41:44.240-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:41:44.247-07:00 INFO UpgradeFromAlice600ITD [main] Running server-add ... 2021-06-09T14:41:44.248-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:41:44.248-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n3.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:41:47.613-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:41:47.624-07:00 INFO UpgradeFromAlice600ITD [main] Rebalancing ... 2021-06-09T14:41:47.624-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:41:47.624-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:41:53.973-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:41:53.974-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:41:53.981-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:41:53.987-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:41:53.987-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:41:53.987-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:41:54.031-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37060] 2021-06-09T14:41:54.032-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:37060 2021-06-09T14:41:54.117-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36628] 2021-06-09T14:41:57.539-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:41:58.235-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:41:58.236-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:41:59.727-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1202 } 2021-06-09T14:41:59.728-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 1202 } 2021-06-09T14:42:00.918-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:42:00.918-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:42:00.918-07:00 INFO UpgradeTestBase [main] Recreating 'n2' node (image = null, forceNewIp = false)... 2021-06-09T14:42:00.919-07:00 INFO DockerTestBase [main] Copying files from docker container a8e4113e6d5f7: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0010_n2_couchbase_enterprise-6.0.0 2021-06-09T14:42:01.114-07:00 INFO DockerTestBase [main] Stopping docker container: a8e4113e6d5f7 2021-06-09T14:42:01.515-07:00 INFO DockerTestBase [main] Stopped docker container: a8e4113e6d5f7 2021-06-09T14:42:01.515-07:00 INFO DockerTestBase [main] Removing docker container: a8e4113e6d5f7 2021-06-09T14:42:01.524-07:00 INFO DockerTestBase [main] Removed docker container: a8e4113e6d5f7 2021-06-09T14:42:01.524-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0012_n2_thisbuild, network = { name = cbas-docker-test-0003-965913b1-4b13-46ea-97dd-9b425ea466fb, id = 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 }) 2021-06-09T14:42:01.525-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:40092/tcp 2021-06-09T14:42:01.525-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:38320/tcp 2021-06-09T14:42:01.570-07:00 INFO DockerTestBase [main] Starting docker container: a2bb3bbe4faf3 2021-06-09T14:42:02.559-07:00 INFO DockerTestBase [main] Started docker container: a2bb3bbe4faf3 2021-06-09T14:42:02.565-07:00 INFO DockerTestBase [main] Container information for a2bb3bbe4faf3: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:38320->192.168.176.3:8095, 172.17.0.1:40092->192.168.176.3:8091 2021-06-09T14:42:12.599-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:42:24.126-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:40092] 2021-06-09T14:42:24.127-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:42:24.641-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2949 } 2021-06-09T14:42:24.642-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 2949 } 2021-06-09T14:42:25.831-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2949 } 2021-06-09T14:42:25.832-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2949 } 2021-06-09T14:42:27.566-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 4236 } 2021-06-09T14:42:27.566-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 4236 } 2021-06-09T14:42:28.726-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:42:28.726-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:42:28.727-07:00 INFO UpgradeTestBase [main] Recreating 'n3' node (image = null, forceNewIp = false)... 2021-06-09T14:42:28.727-07:00 INFO DockerTestBase [main] Copying files from docker container e0be3918e2944: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0011_n3_couchbase_enterprise-6.0.0 2021-06-09T14:42:28.910-07:00 INFO DockerTestBase [main] Stopping docker container: e0be3918e2944 2021-06-09T14:42:29.351-07:00 INFO DockerTestBase [main] Stopped docker container: e0be3918e2944 2021-06-09T14:42:29.352-07:00 INFO DockerTestBase [main] Removing docker container: e0be3918e2944 2021-06-09T14:42:29.360-07:00 INFO DockerTestBase [main] Removed docker container: e0be3918e2944 2021-06-09T14:42:29.361-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0013_n3_thisbuild, network = { name = cbas-docker-test-0003-965913b1-4b13-46ea-97dd-9b425ea466fb, id = 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 }) 2021-06-09T14:42:29.362-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:33500/tcp 2021-06-09T14:42:29.362-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:41122/tcp 2021-06-09T14:42:29.421-07:00 INFO DockerTestBase [main] Starting docker container: 2bb9d75598d7a 2021-06-09T14:42:30.300-07:00 INFO DockerTestBase [main] Started docker container: 2bb9d75598d7a 2021-06-09T14:42:30.308-07:00 INFO DockerTestBase [main] Container information for 2bb9d75598d7a: host: n3.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:41122->192.168.176.4:8095, 172.17.0.1:33500->192.168.176.4:8091 2021-06-09T14:42:40.347-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:42:46.621-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:33500] 2021-06-09T14:42:46.622-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:42:47.596-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:42:47.596-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:42:47.596-07:00 INFO DockerTestBase [main] Copying files from docker container a2bb3bbe4faf3: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0012_n2_thisbuild 2021-06-09T14:42:47.601-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container 6c14fea7b2329: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0009_n1_couchbase_enterprise-6.0.0 2021-06-09T14:42:47.604-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 2bb9d75598d7a: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCNewMixedOffline/0013_n3_thisbuild 2021-06-09T14:42:48.026-07:00 INFO DockerTestBase [main] Stopping docker container: a2bb3bbe4faf3 2021-06-09T14:42:48.102-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: 6c14fea7b2329 2021-06-09T14:42:48.110-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 2bb9d75598d7a 2021-06-09T14:42:48.627-07:00 INFO DockerTestBase [main] Stopped docker container: a2bb3bbe4faf3 2021-06-09T14:42:48.788-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 2bb9d75598d7a 2021-06-09T14:42:48.841-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: 6c14fea7b2329 2021-06-09T14:42:48.843-07:00 INFO DockerTestBase [main] Removing docker container: a2bb3bbe4faf3 2021-06-09T14:42:48.843-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removing docker container: 2bb9d75598d7a 2021-06-09T14:42:48.843-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 6c14fea7b2329 2021-06-09T14:42:48.859-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removed docker container: 2bb9d75598d7a 2021-06-09T14:42:48.863-07:00 INFO DockerTestBase [main] Removed docker container: a2bb3bbe4faf3 2021-06-09T14:42:48.868-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 6c14fea7b2329 2021-06-09T14:42:48.868-07:00 INFO DockerTestBase [main] Removing docker network: 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 2021-06-09T14:42:48.954-07:00 INFO DockerTestBase [main] Removed docker network: 6425fb3327728d1f8b2d87a914970bf5ad70a03acdd9ff7982ec2cca638a7429 2021-06-09T14:42:48.959-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyCCNewMixedOffline SUCCEEDED 2021-06-09T14:42:48.960-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyCCOldMixedOffline START 2021-06-09T14:42:48.964-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:42:49.011-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0004-c789ba42-fd2f-4525-b7de-e75761142110) 2021-06-09T14:42:49.063-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0004-c789ba42-fd2f-4525-b7de-e75761142110, id = 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 } 2021-06-09T14:42:49.064-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #1 ... 2021-06-09T14:42:49.064-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:42:49.119-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:42:49.120-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0014_n1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0004-c789ba42-fd2f-4525-b7de-e75761142110, id = 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 }) 2021-06-09T14:42:49.124-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:42962/tcp 2021-06-09T14:42:49.125-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:44353/tcp 2021-06-09T14:42:49.189-07:00 INFO DockerTestBase [main] Starting docker container: a8113fac690b5 2021-06-09T14:42:50.280-07:00 INFO DockerTestBase [main] Started docker container: a8113fac690b5 2021-06-09T14:42:50.313-07:00 INFO DockerTestBase [main] Container information for a8113fac690b5: host: n1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:44353->192.168.176.2:8095, 172.17.0.1:42962->192.168.176.2:8091 2021-06-09T14:43:00.581-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:43:00.582-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:43:01.185-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:43:01.193-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:43:01.193-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:43:02.248-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:43:02.259-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #2 ... 2021-06-09T14:43:02.260-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:43:02.272-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:43:02.272-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0015_n2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0004-c789ba42-fd2f-4525-b7de-e75761142110, id = 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 }) 2021-06-09T14:43:02.273-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:35350/tcp 2021-06-09T14:43:02.273-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:38952/tcp 2021-06-09T14:43:02.319-07:00 INFO DockerTestBase [main] Starting docker container: ae2d47b227752 2021-06-09T14:43:03.248-07:00 INFO DockerTestBase [main] Started docker container: ae2d47b227752 2021-06-09T14:43:03.264-07:00 INFO DockerTestBase [main] Container information for ae2d47b227752: host: n2.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:38952->192.168.176.3:8095, 172.17.0.1:35350->192.168.176.3:8091 2021-06-09T14:43:14.302-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:43:14.302-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:43:14.944-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:43:14.953-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:43:14.953-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:43:17.942-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:43:17.950-07:00 INFO UpgradeFromAlice600ITD [main] Rebalancing ... 2021-06-09T14:43:17.950-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:43:17.950-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:43:19.283-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:43:19.283-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:43:19.290-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:43:19.295-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:43:19.296-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:43:19.296-07:00 INFO UpgradeFromAlice600ITD [main] Creating node #3 ... 2021-06-09T14:43:19.296-07:00 INFO UpgradeTestBase [main] Creating n3 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:43:19.308-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n3 to /opt/couchbase/var 2021-06-09T14:43:19.309-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0016_n3_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0004-c789ba42-fd2f-4525-b7de-e75761142110, id = 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 }) 2021-06-09T14:43:19.309-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:40287/tcp 2021-06-09T14:43:19.309-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:34713/tcp 2021-06-09T14:43:19.372-07:00 INFO DockerTestBase [main] Starting docker container: 09b60211f5b14 2021-06-09T14:43:20.054-07:00 INFO DockerTestBase [main] Started docker container: 09b60211f5b14 2021-06-09T14:43:20.067-07:00 INFO DockerTestBase [main] Container information for 09b60211f5b14: host: n3.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.4, 172.17.0.1:34713->192.168.176.4:8095, 172.17.0.1:40287->192.168.176.4:8091 2021-06-09T14:43:30.097-07:00 INFO DockerTestBase [main] Running 'node-init' on node n3 (192.168.176.4) 2021-06-09T14:43:30.098-07:00 INFO DockerTestBase [main] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n3.couchbase.host 2021-06-09T14:43:30.612-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:43:30.621-07:00 INFO UpgradeFromAlice600ITD [main] Running server-add ... 2021-06-09T14:43:30.621-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:43:30.621-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n3.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:43:33.860-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:43:33.868-07:00 INFO UpgradeFromAlice600ITD [main] Rebalancing ... 2021-06-09T14:43:33.868-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:43:33.869-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:43:41.246-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:43:41.246-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:43:41.253-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:43:41.257-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:43:41.258-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:43:41.258-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:43:41.298-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42962] 2021-06-09T14:43:41.299-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:42962 2021-06-09T14:43:41.365-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35350] 2021-06-09T14:43:45.748-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:43:46.384-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:43:46.384-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:43:47.774-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1078 } 2021-06-09T14:43:47.774-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 1078 } 2021-06-09T14:43:49.021-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:43:49.021-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:43:49.022-07:00 INFO UpgradeTestBase [main] Recreating 'n3' node (image = null, forceNewIp = false)... 2021-06-09T14:43:49.022-07:00 INFO DockerTestBase [main] Copying files from docker container 09b60211f5b14: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0016_n3_couchbase_enterprise-6.0.0 2021-06-09T14:43:49.292-07:00 INFO DockerTestBase [main] Stopping docker container: 09b60211f5b14 2021-06-09T14:43:49.766-07:00 INFO DockerTestBase [main] Stopped docker container: 09b60211f5b14 2021-06-09T14:43:49.766-07:00 INFO DockerTestBase [main] Removing docker container: 09b60211f5b14 2021-06-09T14:43:49.780-07:00 INFO DockerTestBase [main] Removed docker container: 09b60211f5b14 2021-06-09T14:43:49.780-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0017_n3_thisbuild, network = { name = cbas-docker-test-0004-c789ba42-fd2f-4525-b7de-e75761142110, id = 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 }) 2021-06-09T14:43:49.781-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:44454/tcp 2021-06-09T14:43:49.781-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:39760/tcp 2021-06-09T14:43:49.835-07:00 INFO DockerTestBase [main] Starting docker container: 54089309c2463 2021-06-09T14:43:51.597-07:00 INFO DockerTestBase [main] Started docker container: 54089309c2463 2021-06-09T14:43:51.604-07:00 INFO DockerTestBase [main] Container information for 54089309c2463: host: n3.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:39760->192.168.176.4:8095, 172.17.0.1:44454->192.168.176.4:8091 2021-06-09T14:44:02.639-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:44:10.341-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:44454] 2021-06-09T14:44:10.341-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:44:11.127-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2942 } 2021-06-09T14:44:11.127-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 2942 } 2021-06-09T14:44:12.444-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 4405 } 2021-06-09T14:44:12.444-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 4405 } 2021-06-09T14:44:13.636-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5542 } 2021-06-09T14:44:13.637-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 5542 } 2021-06-09T14:44:14.775-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:44:14.775-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:44:14.775-07:00 INFO UpgradeTestBase [main] Recreating 'n2' node (image = null, forceNewIp = false)... 2021-06-09T14:44:14.775-07:00 INFO DockerTestBase [main] Copying files from docker container ae2d47b227752: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0015_n2_couchbase_enterprise-6.0.0 2021-06-09T14:44:15.147-07:00 INFO DockerTestBase [main] Stopping docker container: ae2d47b227752 2021-06-09T14:44:15.857-07:00 INFO DockerTestBase [main] Stopped docker container: ae2d47b227752 2021-06-09T14:44:15.858-07:00 INFO DockerTestBase [main] Removing docker container: ae2d47b227752 2021-06-09T14:44:15.870-07:00 INFO DockerTestBase [main] Removed docker container: ae2d47b227752 2021-06-09T14:44:15.871-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0018_n2_thisbuild, network = { name = cbas-docker-test-0004-c789ba42-fd2f-4525-b7de-e75761142110, id = 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 }) 2021-06-09T14:44:15.875-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:35938/tcp 2021-06-09T14:44:15.876-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:42405/tcp 2021-06-09T14:44:15.915-07:00 INFO DockerTestBase [main] Starting docker container: 087571291bafd 2021-06-09T14:44:17.501-07:00 INFO DockerTestBase [main] Started docker container: 087571291bafd 2021-06-09T14:44:17.506-07:00 INFO DockerTestBase [main] Container information for 087571291bafd: host: n2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:42405->192.168.176.3:8095, 172.17.0.1:35938->192.168.176.3:8091 2021-06-09T14:44:27.541-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:44:39.099-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35938] 2021-06-09T14:44:39.099-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:44:40.140-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:44:40.140-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:44:40.141-07:00 INFO DockerTestBase [main] Copying files from docker container 54089309c2463: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0017_n3_thisbuild 2021-06-09T14:44:40.142-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 087571291bafd: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0018_n2_thisbuild 2021-06-09T14:44:40.142-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Copying files from docker container a8113fac690b5: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeToCheckoutCBASOnlyCCOldMixedOffline/0014_n1_couchbase_enterprise-6.0.0 2021-06-09T14:44:40.488-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 087571291bafd 2021-06-09T14:44:40.578-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopping docker container: a8113fac690b5 2021-06-09T14:44:40.579-07:00 INFO DockerTestBase [main] Stopping docker container: 54089309c2463 2021-06-09T14:44:41.091-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 087571291bafd 2021-06-09T14:44:41.323-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopped docker container: a8113fac690b5 2021-06-09T14:44:41.362-07:00 INFO DockerTestBase [main] Stopped docker container: 54089309c2463 2021-06-09T14:44:41.363-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: a8113fac690b5 2021-06-09T14:44:41.363-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 087571291bafd 2021-06-09T14:44:41.363-07:00 INFO DockerTestBase [main] Removing docker container: 54089309c2463 2021-06-09T14:44:41.374-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 087571291bafd 2021-06-09T14:44:41.375-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: a8113fac690b5 2021-06-09T14:44:41.379-07:00 INFO DockerTestBase [main] Removed docker container: 54089309c2463 2021-06-09T14:44:41.380-07:00 INFO DockerTestBase [main] Removing docker network: 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 2021-06-09T14:44:41.449-07:00 INFO DockerTestBase [main] Removed docker network: 0356cea3e2b0c1cb7c15b5cd745756c7878faaebd21df6ec0d1140ba603841d0 2021-06-09T14:44:41.452-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyCCOldMixedOffline SUCCEEDED 2021-06-09T14:44:41.453-07:00 WARN TestMethodTracer [main] ### upgradeEnsureTxnLog START 2021-06-09T14:44:41.455-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:44:41.494-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0005-d86b78d5-d337-409b-824d-266a03bd7b5f) 2021-06-09T14:44:41.537-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0005-d86b78d5-d337-409b-824d-266a03bd7b5f, id = 25a5db86b2362eed9f9cea019df3560e823f53deaf595964122e27dcc8444e32 } 2021-06-09T14:44:41.538-07:00 INFO UpgradeTestBase [main] Creating kv node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:44:41.539-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv to /opt/couchbase/var 2021-06-09T14:44:41.540-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0019_kv_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0005-d86b78d5-d337-409b-824d-266a03bd7b5f, id = 25a5db86b2362eed9f9cea019df3560e823f53deaf595964122e27dcc8444e32 }) 2021-06-09T14:44:41.544-07:00 INFO DockerTestBase [main] binding exposed port kv:8091/tcp -> 172.17.0.1:39102/tcp 2021-06-09T14:44:41.545-07:00 INFO DockerTestBase [main] binding exposed port kv:8095/tcp -> 172.17.0.1:43624/tcp 2021-06-09T14:44:41.604-07:00 INFO DockerTestBase [main] Starting docker container: 107dc9c474f1e 2021-06-09T14:44:42.499-07:00 INFO DockerTestBase [main] Started docker container: 107dc9c474f1e 2021-06-09T14:44:42.529-07:00 INFO DockerTestBase [main] Container information for 107dc9c474f1e: host: kv.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:43624->192.168.176.2:8095, 172.17.0.1:39102->192.168.176.2:8091 2021-06-09T14:44:54.582-07:00 INFO DockerTestBase [main] Running 'node-init' on node kv (192.168.176.2) 2021-06-09T14:44:54.582-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv.couchbase.host 2021-06-09T14:44:55.368-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:44:55.376-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node kv (192.168.176.2) with services: [DATA] 2021-06-09T14:44:55.376-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:44:56.426-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:44:56.433-07:00 INFO UpgradeTestBase [main] Creating cbas node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:44:56.434-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002cbas to /opt/couchbase/var 2021-06-09T14:44:56.434-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0020_cbas_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0005-d86b78d5-d337-409b-824d-266a03bd7b5f, id = 25a5db86b2362eed9f9cea019df3560e823f53deaf595964122e27dcc8444e32 }) 2021-06-09T14:44:56.434-07:00 INFO DockerTestBase [main] binding exposed port cbas:8091/tcp -> 172.17.0.1:35485/tcp 2021-06-09T14:44:56.435-07:00 INFO DockerTestBase [main] binding exposed port cbas:8095/tcp -> 172.17.0.1:41281/tcp 2021-06-09T14:44:56.531-07:00 INFO DockerTestBase [main] Starting docker container: 7608376b88eb9 2021-06-09T14:44:58.182-07:00 INFO DockerTestBase [main] Started docker container: 7608376b88eb9 2021-06-09T14:44:58.188-07:00 INFO DockerTestBase [main] Container information for 7608376b88eb9: host: cbas.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:41281->192.168.176.3:8095, 172.17.0.1:35485->192.168.176.3:8091 2021-06-09T14:45:11.253-07:00 INFO DockerTestBase [main] Running 'node-init' on node cbas (192.168.176.3) 2021-06-09T14:45:11.253-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cbas.couchbase.host 2021-06-09T14:45:12.030-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:45:12.039-07:00 INFO DockerTestBase [main] Running 'server-add' for node cbas (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:45:12.040-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:45:16.618-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:45:16.631-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:45:16.631-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:45:20.996-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:45:20.996-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:45:21.003-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:45:21.008-07:00 INFO DockerTestBase [pool-1] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:45:21.008-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:45:21.009-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:45:29.402-07:00 INFO UpgradeFromAlice600ITD [main] Swap-rebalancing KV node ... 2021-06-09T14:45:29.403-07:00 INFO UpgradeTestBase [main] Creating kv-new node using 'this build' docker image... 2021-06-09T14:45:29.403-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003kvnew to /opt/couchbase/var 2021-06-09T14:45:29.403-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0021_kvnew_thisbuild, network = { name = cbas-docker-test-0005-d86b78d5-d337-409b-824d-266a03bd7b5f, id = 25a5db86b2362eed9f9cea019df3560e823f53deaf595964122e27dcc8444e32 }) 2021-06-09T14:45:29.404-07:00 INFO DockerTestBase [main] binding exposed port kvnew:8091/tcp -> 172.17.0.1:44650/tcp 2021-06-09T14:45:29.404-07:00 INFO DockerTestBase [main] binding exposed port kvnew:8095/tcp -> 172.17.0.1:43655/tcp 2021-06-09T14:45:29.439-07:00 INFO DockerTestBase [main] Starting docker container: d3dd411514186 2021-06-09T14:45:31.046-07:00 INFO DockerTestBase [main] Started docker container: d3dd411514186 2021-06-09T14:45:31.051-07:00 INFO DockerTestBase [main] Container information for d3dd411514186: host: kvnew.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:43655->192.168.176.4:8095, 172.17.0.1:44650->192.168.176.4:8091 2021-06-09T14:45:41.232-07:00 INFO DockerTestBase [main] Running 'node-init' on node kvnew (192.168.176.4) 2021-06-09T14:45:41.232-07:00 INFO DockerTestBase [main] Executing the command: docker exec kvnew.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname kvnew.couchbase.host 2021-06-09T14:45:42.140-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:45:42.165-07:00 INFO DockerTestBase [main] Running 'server-add' for node kvnew (192.168.176.4) with services: [DATA] 2021-06-09T14:45:42.166-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://kvnew.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T14:45:45.785-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:45:45.793-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:45:45.795-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase --server-remove kv.couchbase.host 2021-06-09T14:45:59.855-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:45:59.856-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:45:59.863-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:44650] 2021-06-09T14:45:59.864-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:44650 2021-06-09T14:46:01.129-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35485] 2021-06-09T14:46:02.134-07:00 INFO UpgradeFromAlice600ITD [main] Creating magic bucket... 2021-06-09T14:46:02.161-07:00 INFO KvStoreHttpHelper [main] Bucket (magic_bucket) does not exist. 2021-06-09T14:46:02.181-07:00 INFO KvStoreHttpHelper [main] Bucket (magic_bucket) created successfully. 2021-06-09T14:46:02.211-07:00 WARN KvStoreHttpHelper [main] Bucket magic_bucket creation not yet verified; retrying after 2s 2021-06-09T14:46:04.438-07:00 INFO UpgradeFromAlice600ITD [main] Creating magic dataset... 2021-06-09T14:46:05.186-07:00 INFO UpgradeFromAlice600ITD [main] Creating magic index... 2021-06-09T14:46:05.351-07:00 INFO UpgradeTestBase [main] Connecting link... 2021-06-09T14:46:08.517-07:00 INFO UpgradeFromAlice600ITD [main] Waiting for magic bucket to be ingested... 2021-06-09T14:46:09.408-07:00 INFO DockerTestBase [main] ResultSet found : { "cnt": 0 } 2021-06-09T14:46:09.408-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:46:09.408-07:00 INFO UpgradeFromAlice600ITD [main] Ingesting magic nulls into magic bucket... 2021-06-09T14:46:09.456-07:00 INFO UpgradeFromAlice600ITD [main] Waiting for magic bucket to be ingested... 2021-06-09T14:46:09.665-07:00 INFO DockerTestBase [main] ResultSet found : { "cnt": 0 } 2021-06-09T14:46:09.665-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "cnt": 0 } 2021-06-09T14:46:10.745-07:00 INFO DockerTestBase [main] ResultSet found : { "cnt": 3 } 2021-06-09T14:46:10.745-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:46:10.746-07:00 INFO UpgradeFromAlice600ITD [main] Killing node (should ensure txn log) ... 2021-06-09T14:46:10.746-07:00 INFO DockerTestBase [main] Copying files from docker container 7608376b88eb9: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0020_cbas_couchbase_enterprise-6.0.0 2021-06-09T14:46:11.006-07:00 INFO DockerTestBase [main] Stopping docker container: 7608376b88eb9 2021-06-09T14:46:11.433-07:00 INFO DockerTestBase [main] Stopped docker container: 7608376b88eb9 2021-06-09T14:46:11.434-07:00 INFO DockerTestBase [main] Removing docker container: 7608376b88eb9 2021-06-09T14:46:11.444-07:00 INFO DockerTestBase [main] Removed docker container: 7608376b88eb9 2021-06-09T14:46:11.444-07:00 INFO UpgradeFromAlice600ITD [main] Backing up pre-upgrade state ... 2021-06-09T14:46:11.497-07:00 INFO UpgradeFromAlice600ITD [main] Upgrading node ... 2021-06-09T14:46:11.497-07:00 INFO UpgradeTestBase [main] Recreating 'cbas' node (image = null, forceNewIp = false)... 2021-06-09T14:46:11.497-07:00 INFO DockerTestBase [main] Not stopping docker container 7608376b88eb9 (not running) 2021-06-09T14:46:11.497-07:00 INFO DockerTestBase [main] Not removing already removed docker container 7608376b88eb9 2021-06-09T14:46:11.498-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0022_cbas_thisbuild, network = { name = cbas-docker-test-0005-d86b78d5-d337-409b-824d-266a03bd7b5f, id = 25a5db86b2362eed9f9cea019df3560e823f53deaf595964122e27dcc8444e32 }) 2021-06-09T14:46:11.499-07:00 INFO DockerTestBase [main] binding exposed port cbas:8091/tcp -> 172.17.0.1:41698/tcp 2021-06-09T14:46:11.499-07:00 INFO DockerTestBase [main] binding exposed port cbas:8095/tcp -> 172.17.0.1:36660/tcp 2021-06-09T14:46:11.551-07:00 INFO DockerTestBase [main] Starting docker container: 21f1dc3271984 2021-06-09T14:46:12.818-07:00 INFO DockerTestBase [main] Started docker container: 21f1dc3271984 2021-06-09T14:46:12.827-07:00 INFO DockerTestBase [main] Container information for 21f1dc3271984: host: cbas.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:36660->192.168.176.3:8095, 172.17.0.1:41698->192.168.176.3:8091 2021-06-09T14:46:23.864-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE,REBALANCE_REQUIRED} for up to 120s... 2021-06-09T14:46:41.395-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T14:46:41.863-07:00 INFO UpgradeFromAlice600ITD [main] Killing & removing node now that clusterCompat has been updated ... 2021-06-09T14:46:41.864-07:00 INFO DockerTestBase [main] Copying files from docker container 21f1dc3271984: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0022_cbas_thisbuild 2021-06-09T14:46:42.356-07:00 INFO DockerTestBase [main] Stopping docker container: 21f1dc3271984 2021-06-09T14:46:42.732-07:00 INFO DockerTestBase [main] Stopped docker container: 21f1dc3271984 2021-06-09T14:46:42.732-07:00 INFO UpgradeFromAlice600ITD [main] Restoring pre-upgrade state ... 2021-06-09T14:46:42.778-07:00 INFO DockerTestBase [main] Starting docker container: 21f1dc3271984 2021-06-09T14:46:43.949-07:00 INFO DockerTestBase [main] Started docker container: 21f1dc3271984 2021-06-09T14:46:43.956-07:00 INFO DockerTestBase [main] Container information for 21f1dc3271984: host: cbas.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:36660->192.168.176.3:8095, 172.17.0.1:41698->192.168.176.3:8091 2021-06-09T14:46:43.957-07:00 INFO UpgradeFromAlice600ITD [main] Waiting for REBALANCE_REQUIRED ... 2021-06-09T14:46:43.957-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {REBALANCE_REQUIRED} for up to 120s... 2021-06-09T14:47:02.563-07:00 INFO UpgradeFromAlice600ITD [main] Running required rebalance ... 2021-06-09T14:47:02.563-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:47:02.564-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase 2021-06-09T14:47:09.208-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:47:09.209-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:47:09.400-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:47:09.411-07:00 INFO DockerTestBase [pool-1] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:47:09.412-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:47:09.412-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:47:10.583-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:47:34.779-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:41698] 2021-06-09T14:47:36.750-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:47:37.467-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:47:37.467-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:47:37.468-07:00 INFO UpgradeFromAlice600ITD [main] Waiting for magic bucket to be ingested... 2021-06-09T14:47:37.640-07:00 INFO DockerTestBase [main] ResultSet found : { "cnt": 3 } 2021-06-09T14:47:37.640-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:47:37.640-07:00 INFO UpgradeFromAlice600ITD [main] Checking no nulls in secondary index... 2021-06-09T14:47:37.879-07:00 INFO DockerTestBase [main] Copying files from docker container 21f1dc3271984: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0022_cbas_thisbuild 2021-06-09T14:47:38.581-07:00 INFO DockerTestBase [main] Stopping docker container: 21f1dc3271984 2021-06-09T14:47:39.019-07:00 INFO DockerTestBase [main] Stopped docker container: 21f1dc3271984 2021-06-09T14:47:39.034-07:00 INFO DockerTestBase [main] found 3 occurrences of 'Logs REDO phase completed' in analytics_info.log on cbas (192.168.176.3) 2021-06-09T14:47:39.036-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Copying files from docker container 107dc9c474f1e: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0019_kv_couchbase_enterprise-6.0.0 2021-06-09T14:47:39.038-07:00 INFO DockerTestBase [main] Copying files from docker container d3dd411514186: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0021_kvnew_thisbuild 2021-06-09T14:47:39.273-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopping docker container: 107dc9c474f1e 2021-06-09T14:47:39.615-07:00 INFO DockerTestBase [main] Stopping docker container: d3dd411514186 2021-06-09T14:47:39.731-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopped docker container: 107dc9c474f1e 2021-06-09T14:47:40.002-07:00 INFO DockerTestBase [main] Stopped docker container: d3dd411514186 2021-06-09T14:47:40.003-07:00 INFO DockerTestBase [main] Removing docker container: d3dd411514186 2021-06-09T14:47:40.004-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removing docker container: 107dc9c474f1e 2021-06-09T14:47:40.004-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 21f1dc3271984 2021-06-09T14:47:40.014-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 21f1dc3271984 2021-06-09T14:47:40.014-07:00 INFO DockerTestBase [main] Removed docker container: d3dd411514186 2021-06-09T14:47:40.015-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removed docker container: 107dc9c474f1e 2021-06-09T14:47:40.016-07:00 INFO DockerTestBase [main] Removing docker network: 25a5db86b2362eed9f9cea019df3560e823f53deaf595964122e27dcc8444e32 2021-06-09T14:47:40.077-07:00 INFO DockerTestBase [main] Removed docker network: 25a5db86b2362eed9f9cea019df3560e823f53deaf595964122e27dcc8444e32 2021-06-09T14:47:40.079-07:00 WARN TestMethodTracer [main] ### upgradeEnsureTxnLog SUCCEEDED [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 574.035 s - in com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD 2021-06-09 14:47:42,830 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:47:42.908-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:47:42.943-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:47:42.944-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:47:43.442-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.0.0 2021-06-09T14:47:44.605-07:00 INFO DockerTestBase [main+] >> enterprise-6.0.0: Pulling from library/couchbase 2021-06-09T14:47:44.612-07:00 INFO DockerTestBase [main+] >> Digest: sha256:342fb755176fb28c826b705721d69eb0fbba8631fd41f3a103806c235bbc6760 2021-06-09T14:47:44.612-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for couchbase:enterprise-6.0.0 2021-06-09T14:47:44.613-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.0.0 2021-06-09T14:47:44.622-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyOnline START 2021-06-09T14:47:44.658-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:47:45.997-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a) 2021-06-09T14:47:46.067-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a, id = 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf } 2021-06-09T14:47:46.069-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:47:46.073-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:47:46.078-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0001_n1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a, id = 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf }) 2021-06-09T14:47:46.103-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:44253/tcp 2021-06-09T14:47:46.104-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:41334/tcp 2021-06-09T14:47:46.227-07:00 INFO DockerTestBase [main] Starting docker container: 9a31b1e4199ac 2021-06-09T14:47:46.850-07:00 ERRO DockerTestBase [main] Error starting docker container: 9a31b1e4199ac com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: driver failed programming external connectivity on endpoint n1.couchbase.host (df2c535d270535f3b3639c40b586339f25218b4e077ef5a0444d69f521beb723): Error starting userland proxy: listen tcp 0.0.0.0:41334: bind: address already in use at com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:68) ~[docker-java-transport-jersey-3.2.8.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608) ~[jersey-client-2.30.1.jar:?] at com.github.dockerjava.jaxrs.JerseyDockerHttpClient.execute(JerseyDockerHttpClient.java:297) ~[docker-java-transport-jersey-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:102) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:31) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:13) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.StartContainerCmdImpl.exec(StartContainerCmdImpl.java:43) ~[docker-java-core-3.2.8.jar:?] at com.couchbase.analytics.test.docker.DockerTestBase.startContainer(DockerTestBase.java:631) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:567) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:453) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:428) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:421) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:415) [test-classes/:?] at com.couchbase.analytics.test.docker.upgrade.UpgradeTestBase.createNode(UpgradeTestBase.java:156) [test-classes/:?] at com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD.upgradeToCheckoutCBASOnlyOnline(UpgradeFromAlice600OnlineITD.java:34) [test-classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) [junit-4.13.jar:4.13] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.13.jar:4.13] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.jar:4.13] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) [surefire-booter-2.21.0.jar:2.21.0] 2021-06-09T14:47:46.865-07:00 WARN DockerTestBase [main] failed to start container n1 (null); will attempt up to 5 times, e java.io.IOException: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: driver failed programming external connectivity on endpoint n1.couchbase.host (df2c535d270535f3b3639c40b586339f25218b4e077ef5a0444d69f521beb723): Error starting userland proxy: listen tcp 0.0.0.0:41334: bind: address already in use at com.couchbase.analytics.test.docker.DockerTestBase.startContainer(DockerTestBase.java:647) ~[test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:567) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:453) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:428) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:421) [test-classes/:?] at com.couchbase.analytics.test.docker.DockerTestBase.createAndStartContainer(DockerTestBase.java:415) [test-classes/:?] at com.couchbase.analytics.test.docker.upgrade.UpgradeTestBase.createNode(UpgradeTestBase.java:156) [test-classes/:?] at com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD.upgradeToCheckoutCBASOnlyOnline(UpgradeFromAlice600OnlineITD.java:34) [test-classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) [junit-4.13.jar:4.13] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.13.jar:4.13] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) [junit-4.13.jar:4.13] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) [junit-4.13.jar:4.13] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) [junit-4.13.jar:4.13] at org.junit.runners.ParentRunner.run(ParentRunner.java:413) [junit-4.13.jar:4.13] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) [surefire-junit4-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125) [surefire-booter-2.21.0.jar:2.21.0] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413) [surefire-booter-2.21.0.jar:2.21.0] Caused by: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: driver failed programming external connectivity on endpoint n1.couchbase.host (df2c535d270535f3b3639c40b586339f25218b4e077ef5a0444d69f521beb723): Error starting userland proxy: listen tcp 0.0.0.0:41334: bind: address already in use at com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:68) ~[docker-java-transport-jersey-3.2.8.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:171) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:159) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:301) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:609) ~[jersey-client-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[jersey-common-2.30.1.jar:?] at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:608) ~[jersey-client-2.30.1.jar:?] at com.github.dockerjava.jaxrs.JerseyDockerHttpClient.execute(JerseyDockerHttpClient.java:297) ~[docker-java-transport-jersey-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:102) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:31) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:13) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35) ~[docker-java-core-3.2.8.jar:?] at com.github.dockerjava.core.command.StartContainerCmdImpl.exec(StartContainerCmdImpl.java:43) ~[docker-java-core-3.2.8.jar:?] at com.couchbase.analytics.test.docker.DockerTestBase.startContainer(DockerTestBase.java:631) ~[test-classes/:?] ... 40 more 2021-06-09T14:47:46.867-07:00 INFO DockerTestBase [main] Removing docker container: 9a31b1e4199ac 2021-06-09T14:47:46.877-07:00 INFO DockerTestBase [main] Removed docker container: 9a31b1e4199ac 2021-06-09T14:47:46.878-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0001_n1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a, id = 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf }) 2021-06-09T14:47:46.878-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:36268/tcp 2021-06-09T14:47:46.879-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:39290/tcp 2021-06-09T14:47:46.928-07:00 INFO DockerTestBase [main] Starting docker container: 8da4843f88d49 2021-06-09T14:47:48.460-07:00 INFO DockerTestBase [main] Started docker container: 8da4843f88d49 2021-06-09T14:47:48.608-07:00 INFO DockerTestBase [main] Container information for 8da4843f88d49: host: n1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:39290->192.168.176.2:8095, 172.17.0.1:36268->192.168.176.2:8091 2021-06-09T14:48:01.719-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:48:01.721-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:48:02.476-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:48:02.487-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:48:02.488-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:48:03.535-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:48:03.544-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:48:03.544-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:48:03.545-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0002_n2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a, id = 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf }) 2021-06-09T14:48:03.546-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:46087/tcp 2021-06-09T14:48:03.546-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:34235/tcp 2021-06-09T14:48:03.599-07:00 INFO DockerTestBase [main] Starting docker container: 2302c51ad7bd9 2021-06-09T14:48:05.162-07:00 INFO DockerTestBase [main] Started docker container: 2302c51ad7bd9 2021-06-09T14:48:05.171-07:00 INFO DockerTestBase [main] Container information for 2302c51ad7bd9: host: n2.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:34235->192.168.176.3:8095, 172.17.0.1:46087->192.168.176.3:8091 2021-06-09T14:48:16.297-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:48:16.297-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:48:16.941-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:48:16.949-07:00 INFO UpgradeTestBase [main] Creating n3 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:48:16.950-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n3 to /opt/couchbase/var 2021-06-09T14:48:16.950-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0003_n3_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a, id = 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf }) 2021-06-09T14:48:16.951-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:35373/tcp 2021-06-09T14:48:16.951-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:37422/tcp 2021-06-09T14:48:16.996-07:00 INFO DockerTestBase [main] Starting docker container: b1f6665c46490 2021-06-09T14:48:18.584-07:00 INFO DockerTestBase [main] Started docker container: b1f6665c46490 2021-06-09T14:48:18.592-07:00 INFO DockerTestBase [main] Container information for b1f6665c46490: host: n3.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.4, 172.17.0.1:37422->192.168.176.4:8095, 172.17.0.1:35373->192.168.176.4:8091 2021-06-09T14:48:30.641-07:00 INFO DockerTestBase [main] Running 'node-init' on node n3 (192.168.176.4) 2021-06-09T14:48:30.642-07:00 INFO DockerTestBase [main] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n3.couchbase.host 2021-06-09T14:48:31.353-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:48:31.361-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:48:31.362-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:48:35.216-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:48:35.223-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:48:35.223-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n3.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:48:38.587-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:48:38.596-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:48:38.597-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:48:39.942-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:48:39.943-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:48:39.961-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:48:39.980-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:48:39.980-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:48:39.998-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:48:46.842-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36268] 2021-06-09T14:48:46.846-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:36268 2021-06-09T14:48:46.930-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46087] 2021-06-09T14:48:50.584-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:48:51.083-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:48:51.089-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:48:52.422-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1058 } 2021-06-09T14:48:52.422-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 1058 } 2021-06-09T14:48:53.715-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3624 } 2021-06-09T14:48:53.715-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 3624 } 2021-06-09T14:48:54.895-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:48:54.896-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:48:54.896-07:00 INFO UpgradeTestBase [main] Creating n2-new node using 'this build' docker image... 2021-06-09T14:48:54.897-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n2new to /opt/couchbase/var 2021-06-09T14:48:54.897-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:48:54.987-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T14:48:54.989-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T14:48:54.989-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:48:54.990-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T14:48:54.996-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0004_n2new_thisbuild, network = { name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a, id = 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf }) 2021-06-09T14:48:54.997-07:00 INFO DockerTestBase [main] binding exposed port n2new:8091/tcp -> 172.17.0.1:46436/tcp 2021-06-09T14:48:54.998-07:00 INFO DockerTestBase [main] binding exposed port n2new:8095/tcp -> 172.17.0.1:37257/tcp 2021-06-09T14:48:55.048-07:00 INFO DockerTestBase [main] Starting docker container: e90dcb95c8776 2021-06-09T14:48:56.148-07:00 INFO DockerTestBase [main] Started docker container: e90dcb95c8776 2021-06-09T14:48:56.169-07:00 INFO DockerTestBase [main] Container information for e90dcb95c8776: host: n2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.5, 172.17.0.1:37257->192.168.176.5:8095, 172.17.0.1:46436->192.168.176.5:8091 2021-06-09T14:49:06.301-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2new (192.168.176.5) 2021-06-09T14:49:06.302-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname n2new.couchbase.host 2021-06-09T14:49:07.324-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:49:07.364-07:00 INFO UpgradeTestBase [main] Creating n3-new node using 'this build' docker image... 2021-06-09T14:49:07.365-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0005n3new to /opt/couchbase/var 2021-06-09T14:49:07.365-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0005_n3new_thisbuild, network = { name = cbas-docker-test-0001-12ec410d-22a7-47c1-ad31-ebfa6128550a, id = 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf }) 2021-06-09T14:49:07.366-07:00 INFO DockerTestBase [main] binding exposed port n3new:8091/tcp -> 172.17.0.1:38097/tcp 2021-06-09T14:49:07.366-07:00 INFO DockerTestBase [main] binding exposed port n3new:8095/tcp -> 172.17.0.1:44301/tcp 2021-06-09T14:49:07.414-07:00 INFO DockerTestBase [main] Starting docker container: 9506902d60d4d 2021-06-09T14:49:08.878-07:00 INFO DockerTestBase [main] Started docker container: 9506902d60d4d 2021-06-09T14:49:08.886-07:00 INFO DockerTestBase [main] Container information for 9506902d60d4d: host: n3new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.6, 172.17.0.1:44301->192.168.176.6:8095, 172.17.0.1:38097->192.168.176.6:8091 2021-06-09T14:49:18.117-07:00 INFO DockerTestBase [main] Running 'node-init' on node n3new (192.168.176.6) 2021-06-09T14:49:18.118-07:00 INFO DockerTestBase [main] Executing the command: docker exec n3new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.6:8091 -u couchbase -p couchbase --node-init-hostname n3new.couchbase.host 2021-06-09T14:49:19.098-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:49:19.120-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2new (192.168.176.5) with services: [ANALYTICS] 2021-06-09T14:49:19.120-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:49:23.601-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:49:23.609-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3new (192.168.176.6) with services: [ANALYTICS] 2021-06-09T14:49:23.610-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n3new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:49:28.148-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:49:28.160-07:00 INFO UpgradeFromAlice600OnlineITD [main] Swap rebalancing ... 2021-06-09T14:49:28.160-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:49:28.162-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase --server-remove n2.couchbase.host,n3.couchbase.host 2021-06-09T14:49:47.861-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:49:47.861-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:49:47.874-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:49:47.874-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:50:00.851-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46436] 2021-06-09T14:50:00.851-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:38097] 2021-06-09T14:50:00.851-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:50:00.851-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T14:50:01.829-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T14:50:01.829-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T14:50:01.840-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:50:01.840-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:50:01.844-07:00 INFO DockerTestBase [main] Copying files from docker container b1f6665c46490: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0003_n3_couchbase_enterprise-6.0.0 2021-06-09T14:50:01.848-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 2302c51ad7bd9: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0002_n2_couchbase_enterprise-6.0.0 2021-06-09T14:50:01.848-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container 9506902d60d4d: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0005_n3new_thisbuild 2021-06-09T14:50:01.849-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Copying files from docker container 8da4843f88d49: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0001_n1_couchbase_enterprise-6.0.0 2021-06-09T14:50:02.257-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: 9506902d60d4d 2021-06-09T14:50:02.326-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopping docker container: 8da4843f88d49 2021-06-09T14:50:02.328-07:00 INFO DockerTestBase [main] Stopping docker container: b1f6665c46490 2021-06-09T14:50:02.357-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 2302c51ad7bd9 2021-06-09T14:50:02.951-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: 9506902d60d4d 2021-06-09T14:50:02.951-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container e90dcb95c8776: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyOnline/0004_n2new_thisbuild 2021-06-09T14:50:03.159-07:00 INFO DockerTestBase [main] Stopped docker container: b1f6665c46490 2021-06-09T14:50:03.191-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: e90dcb95c8776 2021-06-09T14:50:03.205-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopped docker container: 8da4843f88d49 2021-06-09T14:50:03.275-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 2302c51ad7bd9 2021-06-09T14:50:03.582-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: e90dcb95c8776 2021-06-09T14:50:03.584-07:00 INFO DockerTestBase [main] Removing docker container: b1f6665c46490 2021-06-09T14:50:03.584-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 2302c51ad7bd9 2021-06-09T14:50:03.584-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 9506902d60d4d 2021-06-09T14:50:03.584-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 8da4843f88d49 2021-06-09T14:50:03.610-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 8da4843f88d49 2021-06-09T14:50:03.611-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: e90dcb95c8776 2021-06-09T14:50:03.622-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 9506902d60d4d 2021-06-09T14:50:03.622-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 2302c51ad7bd9 2021-06-09T14:50:03.631-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: e90dcb95c8776 2021-06-09T14:50:03.631-07:00 INFO DockerTestBase [main] Removed docker container: b1f6665c46490 2021-06-09T14:50:03.632-07:00 INFO DockerTestBase [main] Removing docker network: 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf 2021-06-09T14:50:03.697-07:00 INFO DockerTestBase [main] Removed docker network: 97af304666ad5897f35a0c9ffc9cbe76a971445937e6575371fb09659f2c5caf 2021-06-09T14:50:03.703-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyOnline SUCCEEDED 2021-06-09T14:50:03.705-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyMixedOnline START 2021-06-09T14:50:03.706-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:50:03.752-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0002-03fdf49e-01f7-46f4-a855-c96131947e7d) 2021-06-09T14:50:03.791-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0002-03fdf49e-01f7-46f4-a855-c96131947e7d, id = 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 } 2021-06-09T14:50:03.791-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:50:03.842-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:50:03.843-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0006_n1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0002-03fdf49e-01f7-46f4-a855-c96131947e7d, id = 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 }) 2021-06-09T14:50:03.846-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:46516/tcp 2021-06-09T14:50:03.847-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:37924/tcp 2021-06-09T14:50:03.913-07:00 INFO DockerTestBase [main] Starting docker container: 0a2a34245679d 2021-06-09T14:50:05.929-07:00 INFO DockerTestBase [main] Started docker container: 0a2a34245679d 2021-06-09T14:50:05.967-07:00 INFO DockerTestBase [main] Container information for 0a2a34245679d: host: n1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:37924->192.168.176.2:8095, 172.17.0.1:46516->192.168.176.2:8091 2021-06-09T14:50:17.009-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T14:50:17.010-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T14:50:17.660-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:50:17.667-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA] 2021-06-09T14:50:17.667-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:50:18.723-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:50:18.730-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:50:18.740-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T14:50:18.740-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0007_n2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0002-03fdf49e-01f7-46f4-a855-c96131947e7d, id = 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 }) 2021-06-09T14:50:18.741-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:43539/tcp 2021-06-09T14:50:18.741-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:44625/tcp 2021-06-09T14:50:18.786-07:00 INFO DockerTestBase [main] Starting docker container: bbfb1d9815a7d 2021-06-09T14:50:19.838-07:00 INFO DockerTestBase [main] Started docker container: bbfb1d9815a7d 2021-06-09T14:50:19.855-07:00 INFO DockerTestBase [main] Container information for bbfb1d9815a7d: host: n2.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:44625->192.168.176.3:8095, 172.17.0.1:43539->192.168.176.3:8091 2021-06-09T14:50:30.140-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T14:50:30.141-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T14:50:30.831-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:50:30.841-07:00 INFO UpgradeTestBase [main] Creating n3 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:50:30.852-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n3 to /opt/couchbase/var 2021-06-09T14:50:30.853-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0008_n3_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0002-03fdf49e-01f7-46f4-a855-c96131947e7d, id = 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 }) 2021-06-09T14:50:30.854-07:00 INFO DockerTestBase [main] binding exposed port n3:8091/tcp -> 172.17.0.1:46553/tcp 2021-06-09T14:50:30.854-07:00 INFO DockerTestBase [main] binding exposed port n3:8095/tcp -> 172.17.0.1:35427/tcp 2021-06-09T14:50:30.907-07:00 INFO DockerTestBase [main] Starting docker container: 8776cd834c4b2 2021-06-09T14:50:31.892-07:00 INFO DockerTestBase [main] Started docker container: 8776cd834c4b2 2021-06-09T14:50:31.899-07:00 INFO DockerTestBase [main] Container information for 8776cd834c4b2: host: n3.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.4, 172.17.0.1:35427->192.168.176.4:8095, 172.17.0.1:46553->192.168.176.4:8091 2021-06-09T14:50:42.034-07:00 INFO DockerTestBase [main] Running 'node-init' on node n3 (192.168.176.4) 2021-06-09T14:50:42.035-07:00 INFO DockerTestBase [main] Executing the command: docker exec n3.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n3.couchbase.host 2021-06-09T14:50:42.580-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:50:42.587-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:50:42.587-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:50:45.705-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:50:45.713-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:50:45.713-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n3.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:50:48.921-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:50:48.928-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:50:48.928-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:50:50.292-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:50:50.293-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:50:50.298-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:50:50.304-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:50:50.304-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:50:50.304-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:50:56.827-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46516] 2021-06-09T14:50:56.828-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:46516 2021-06-09T14:50:56.893-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:43539] 2021-06-09T14:51:00.166-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:51:00.671-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:51:00.671-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:51:02.048-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 978 } 2021-06-09T14:51:02.049-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 978 } 2021-06-09T14:51:03.368-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3483 } 2021-06-09T14:51:03.368-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 3483 } 2021-06-09T14:51:04.499-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:51:04.499-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:51:04.499-07:00 INFO UpgradeTestBase [main] Creating n2-new node using 'this build' docker image... 2021-06-09T14:51:04.513-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n2new to /opt/couchbase/var 2021-06-09T14:51:04.514-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0009_n2new_thisbuild, network = { name = cbas-docker-test-0002-03fdf49e-01f7-46f4-a855-c96131947e7d, id = 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 }) 2021-06-09T14:51:04.514-07:00 INFO DockerTestBase [main] binding exposed port n2new:8091/tcp -> 172.17.0.1:32793/tcp 2021-06-09T14:51:04.515-07:00 INFO DockerTestBase [main] binding exposed port n2new:8095/tcp -> 172.17.0.1:36064/tcp 2021-06-09T14:51:04.602-07:00 INFO DockerTestBase [main] Starting docker container: 63d7a6e1ea3fb 2021-06-09T14:51:05.616-07:00 INFO DockerTestBase [main] Started docker container: 63d7a6e1ea3fb 2021-06-09T14:51:05.632-07:00 INFO DockerTestBase [main] Container information for 63d7a6e1ea3fb: host: n2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.5, 172.17.0.1:36064->192.168.176.5:8095, 172.17.0.1:32793->192.168.176.5:8091 2021-06-09T14:51:15.685-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2new (192.168.176.5) 2021-06-09T14:51:15.685-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname n2new.couchbase.host 2021-06-09T14:51:16.565-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:51:16.597-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2new (192.168.176.5) with services: [ANALYTICS] 2021-06-09T14:51:16.598-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:51:20.599-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:51:20.605-07:00 INFO UpgradeFromAlice600OnlineITD [main] Swap rebalancing ... 2021-06-09T14:51:20.605-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:51:20.606-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase --server-remove n2.couchbase.host 2021-06-09T14:51:41.921-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:51:41.921-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:51:41.933-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:51:41.995-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:32793] 2021-06-09T14:51:41.996-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:51:42.353-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:51:42.353-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:51:42.354-07:00 INFO UpgradeTestBase [main] Creating n3-new node using 'this build' docker image... 2021-06-09T14:51:42.369-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0005n3new to /opt/couchbase/var 2021-06-09T14:51:42.370-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0010_n3new_thisbuild, network = { name = cbas-docker-test-0002-03fdf49e-01f7-46f4-a855-c96131947e7d, id = 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 }) 2021-06-09T14:51:42.371-07:00 INFO DockerTestBase [main] binding exposed port n3new:8091/tcp -> 172.17.0.1:40934/tcp 2021-06-09T14:51:42.371-07:00 INFO DockerTestBase [main] binding exposed port n3new:8095/tcp -> 172.17.0.1:43310/tcp 2021-06-09T14:51:42.428-07:00 INFO DockerTestBase [main] Starting docker container: aeafac1a33e9a 2021-06-09T14:51:43.377-07:00 INFO DockerTestBase [main] Started docker container: aeafac1a33e9a 2021-06-09T14:51:43.384-07:00 INFO DockerTestBase [main] Container information for aeafac1a33e9a: host: n3new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.6, 172.17.0.1:43310->192.168.176.6:8095, 172.17.0.1:40934->192.168.176.6:8091 2021-06-09T14:51:53.425-07:00 INFO DockerTestBase [main] Running 'node-init' on node n3new (192.168.176.6) 2021-06-09T14:51:53.426-07:00 INFO DockerTestBase [main] Executing the command: docker exec n3new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.6:8091 -u couchbase -p couchbase --node-init-hostname n3new.couchbase.host 2021-06-09T14:51:54.226-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:51:54.249-07:00 INFO DockerTestBase [main] Running 'server-add' for node n3new (192.168.176.6) with services: [ANALYTICS] 2021-06-09T14:51:54.250-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n3new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:51:58.204-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:51:58.212-07:00 INFO UpgradeFromAlice600OnlineITD [main] Swap rebalancing ... 2021-06-09T14:51:58.213-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:51:58.213-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase --server-remove n3.couchbase.host 2021-06-09T14:52:07.555-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:52:07.555-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:52:07.562-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:52:19.207-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:40934] 2021-06-09T14:52:19.207-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:52:19.748-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:52:19.748-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:52:19.751-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container bbfb1d9815a7d: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0007_n2_couchbase_enterprise-6.0.0 2021-06-09T14:52:19.751-07:00 INFO DockerTestBase [main] Copying files from docker container 8776cd834c4b2: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0008_n3_couchbase_enterprise-6.0.0 2021-06-09T14:52:19.751-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Copying files from docker container aeafac1a33e9a: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0010_n3new_thisbuild 2021-06-09T14:52:19.755-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container 63d7a6e1ea3fb: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0009_n2new_thisbuild 2021-06-09T14:52:20.074-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: bbfb1d9815a7d 2021-06-09T14:52:20.100-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopping docker container: aeafac1a33e9a 2021-06-09T14:52:20.117-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: 63d7a6e1ea3fb 2021-06-09T14:52:20.207-07:00 INFO DockerTestBase [main] Stopping docker container: 8776cd834c4b2 2021-06-09T14:52:20.789-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: bbfb1d9815a7d 2021-06-09T14:52:20.789-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 0a2a34245679d: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeToCheckoutCBASOnlyMixedOnline/0006_n1_couchbase_enterprise-6.0.0 2021-06-09T14:52:21.080-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: 63d7a6e1ea3fb 2021-06-09T14:52:21.102-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 0a2a34245679d 2021-06-09T14:52:21.133-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopped docker container: aeafac1a33e9a 2021-06-09T14:52:21.168-07:00 INFO DockerTestBase [main] Stopped docker container: 8776cd834c4b2 2021-06-09T14:52:21.507-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 0a2a34245679d 2021-06-09T14:52:21.508-07:00 INFO DockerTestBase [main] Removing docker container: 8776cd834c4b2 2021-06-09T14:52:21.509-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: bbfb1d9815a7d 2021-06-09T14:52:21.509-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: aeafac1a33e9a 2021-06-09T14:52:21.512-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removing docker container: 63d7a6e1ea3fb 2021-06-09T14:52:21.528-07:00 INFO DockerTestBase [main] Removed docker container: 8776cd834c4b2 2021-06-09T14:52:21.541-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removed docker container: 63d7a6e1ea3fb 2021-06-09T14:52:21.541-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removing docker container: 0a2a34245679d 2021-06-09T14:52:21.547-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: aeafac1a33e9a 2021-06-09T14:52:21.558-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: bbfb1d9815a7d 2021-06-09T14:52:21.563-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removed docker container: 0a2a34245679d 2021-06-09T14:52:21.563-07:00 INFO DockerTestBase [main] Removing docker network: 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 2021-06-09T14:52:21.613-07:00 INFO DockerTestBase [main] Removed docker network: 58503c8069e183b48cbcf9b882b6d4249bcc7eba5948752c454c0497a899f9a1 2021-06-09T14:52:21.615-07:00 WARN TestMethodTracer [main] ### upgradeToCheckoutCBASOnlyMixedOnline SUCCEEDED 2021-06-09T14:52:21.615-07:00 WARN TestMethodTracer [main] ### upgradeVerifyClusterCompat START 2021-06-09T14:52:21.617-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:52:21.658-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0003-bc53e64c-7d4f-45c2-96a1-75d611b8307e) 2021-06-09T14:52:21.698-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0003-bc53e64c-7d4f-45c2-96a1-75d611b8307e, id = a6c4d36c14d569e514bca8c0aaa823ebedba7d9ec55aef77e6b048030f1c0613 } 2021-06-09T14:52:21.699-07:00 INFO UpgradeTestBase [main] Creating kv node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:52:21.699-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv to /opt/couchbase/var 2021-06-09T14:52:21.700-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0011_kv_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-bc53e64c-7d4f-45c2-96a1-75d611b8307e, id = a6c4d36c14d569e514bca8c0aaa823ebedba7d9ec55aef77e6b048030f1c0613 }) 2021-06-09T14:52:21.702-07:00 INFO DockerTestBase [main] binding exposed port kv:8091/tcp -> 172.17.0.1:42635/tcp 2021-06-09T14:52:21.702-07:00 INFO DockerTestBase [main] binding exposed port kv:8095/tcp -> 172.17.0.1:41091/tcp 2021-06-09T14:52:21.768-07:00 INFO DockerTestBase [main] Starting docker container: 5b659228ba60a 2021-06-09T14:52:22.806-07:00 INFO DockerTestBase [main] Started docker container: 5b659228ba60a 2021-06-09T14:52:22.842-07:00 INFO DockerTestBase [main] Container information for 5b659228ba60a: host: kv.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.2, 172.17.0.1:41091->192.168.176.2:8095, 172.17.0.1:42635->192.168.176.2:8091 2021-06-09T14:52:33.895-07:00 INFO DockerTestBase [main] Running 'node-init' on node kv (192.168.176.2) 2021-06-09T14:52:33.895-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv.couchbase.host 2021-06-09T14:52:34.656-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:52:34.664-07:00 INFO UpgradeTestBase [main] Creating cbas1 node ([couchbase:enterprise-6.0.0])... 2021-06-09T14:52:34.664-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002cbas1 to /opt/couchbase/var 2021-06-09T14:52:34.664-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0012_cbas1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-bc53e64c-7d4f-45c2-96a1-75d611b8307e, id = a6c4d36c14d569e514bca8c0aaa823ebedba7d9ec55aef77e6b048030f1c0613 }) 2021-06-09T14:52:34.665-07:00 INFO DockerTestBase [main] binding exposed port cbas1:8091/tcp -> 172.17.0.1:37453/tcp 2021-06-09T14:52:34.665-07:00 INFO DockerTestBase [main] binding exposed port cbas1:8095/tcp -> 172.17.0.1:33598/tcp 2021-06-09T14:52:34.735-07:00 INFO DockerTestBase [main] Starting docker container: 5ef666f039d38 2021-06-09T14:52:35.649-07:00 INFO DockerTestBase [main] Started docker container: 5ef666f039d38 2021-06-09T14:52:35.657-07:00 INFO DockerTestBase [main] Container information for 5ef666f039d38: host: cbas1.couchbase.host, image: enterprise-6.0.0, ip: 192.168.176.3, 172.17.0.1:33598->192.168.176.3:8095, 172.17.0.1:37453->192.168.176.3:8091 2021-06-09T14:52:46.706-07:00 INFO DockerTestBase [main] Running 'node-init' on node cbas1 (192.168.176.3) 2021-06-09T14:52:46.706-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cbas1.couchbase.host 2021-06-09T14:52:47.368-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T14:52:47.377-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node kv (192.168.176.2) with services: [DATA] 2021-06-09T14:52:47.377-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T14:52:47.713-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T14:52:47.721-07:00 INFO DockerTestBase [main] Running 'server-add' for node cbas1 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T14:52:47.721-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas1.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:52:51.556-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T14:52:51.563-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T14:52:51.564-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T14:52:55.980-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:52:55.980-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:52:55.988-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:52:55.994-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:52:55.994-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:52:55.995-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:53:03.112-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42635] 2021-06-09T14:53:03.113-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:42635 2021-06-09T14:53:03.164-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:37453] 2021-06-09T14:53:06.888-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:53:07.201-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T14:53:07.201-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T14:53:08.608-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3185 } 2021-06-09T14:53:08.608-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 3185 } 2021-06-09T14:53:09.772-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:53:09.772-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:53:09.773-07:00 INFO UpgradeTestBase [main] Upgrading [cbas1.couchbase.host] (online, rebalance) ... 2021-06-09T14:53:09.775-07:00 INFO UpgradeTestBase [pool-0] Creating cbas1-new node using 'this build' docker image... 2021-06-09T14:53:09.775-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003cbas1new to /opt/couchbase/var 2021-06-09T14:53:09.776-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0013_cbas1new_thisbuild, network = { name = cbas-docker-test-0003-bc53e64c-7d4f-45c2-96a1-75d611b8307e, id = a6c4d36c14d569e514bca8c0aaa823ebedba7d9ec55aef77e6b048030f1c0613 }) 2021-06-09T14:53:09.777-07:00 INFO DockerTestBase [pool-0] binding exposed port cbas1new:8091/tcp -> 172.17.0.1:46001/tcp 2021-06-09T14:53:09.777-07:00 INFO DockerTestBase [pool-0] binding exposed port cbas1new:8095/tcp -> 172.17.0.1:45055/tcp 2021-06-09T14:53:09.852-07:00 INFO DockerTestBase [pool-0] Starting docker container: 76ea0226c5c98 2021-06-09T14:53:10.881-07:00 INFO DockerTestBase [pool-0] Started docker container: 76ea0226c5c98 2021-06-09T14:53:10.887-07:00 INFO DockerTestBase [pool-0] Container information for 76ea0226c5c98: host: cbas1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:45055->192.168.176.4:8095, 172.17.0.1:46001->192.168.176.4:8091 2021-06-09T14:53:20.939-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node cbas1new (192.168.176.4) 2021-06-09T14:53:20.939-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec cbas1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname cbas1new.couchbase.host 2021-06-09T14:53:21.697-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:53:21.725-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node cbas1new (192.168.176.4) with services: [ANALYTICS] 2021-06-09T14:53:21.726-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T14:53:25.955-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T14:53:25.964-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T14:53:25.964-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:53:25.964-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase --server-remove cbas1.couchbase.host 2021-06-09T14:53:46.227-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:53:46.228-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:53:46.234-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:53:46.242-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:53:46.243-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:53:46.244-07:00 INFO DockerTestBase [main] Copying files from docker container 5ef666f039d38: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0012_cbas1_couchbase_enterprise-6.0.0 2021-06-09T14:53:46.555-07:00 INFO DockerTestBase [main] Stopping docker container: 5ef666f039d38 2021-06-09T14:53:46.985-07:00 INFO DockerTestBase [main] Stopped docker container: 5ef666f039d38 2021-06-09T14:53:46.985-07:00 INFO DockerTestBase [main] Removing docker container: 5ef666f039d38 2021-06-09T14:53:46.993-07:00 INFO DockerTestBase [main] Removed docker container: 5ef666f039d38 2021-06-09T14:53:46.993-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:53:54.814-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46001] 2021-06-09T14:53:54.815-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:53:55.389-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:53:55.389-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:53:55.531-07:00 INFO DockerTestBase [main] assuming 6.0 compatibility on missing 'clusterCompatibility' field 2021-06-09T14:53:55.532-07:00 INFO UpgradeTestBase [main] Upgrading [kv.couchbase.host] (online, rebalance) ... 2021-06-09T14:53:55.532-07:00 INFO UpgradeTestBase [pool-0] Creating kv-new node using 'this build' docker image... 2021-06-09T14:53:55.533-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004kvnew to /opt/couchbase/var 2021-06-09T14:53:55.534-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0014_kvnew_thisbuild, network = { name = cbas-docker-test-0003-bc53e64c-7d4f-45c2-96a1-75d611b8307e, id = a6c4d36c14d569e514bca8c0aaa823ebedba7d9ec55aef77e6b048030f1c0613 }) 2021-06-09T14:53:55.534-07:00 INFO DockerTestBase [pool-0] binding exposed port kvnew:8091/tcp -> 172.17.0.1:43475/tcp 2021-06-09T14:53:55.534-07:00 INFO DockerTestBase [pool-0] binding exposed port kvnew:8095/tcp -> 172.17.0.1:35215/tcp 2021-06-09T14:53:55.639-07:00 INFO DockerTestBase [pool-0] Starting docker container: dcaf57bb602ec 2021-06-09T14:53:56.789-07:00 INFO DockerTestBase [pool-0] Started docker container: dcaf57bb602ec 2021-06-09T14:53:56.800-07:00 INFO DockerTestBase [pool-0] Container information for dcaf57bb602ec: host: kvnew.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:35215->192.168.176.3:8095, 172.17.0.1:43475->192.168.176.3:8091 2021-06-09T14:54:06.148-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node kvnew (192.168.176.3) 2021-06-09T14:54:06.148-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kvnew.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname kvnew.couchbase.host 2021-06-09T14:54:07.001-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T14:54:07.028-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node kvnew (192.168.176.3) with services: [DATA] 2021-06-09T14:54:07.028-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec cbas1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://kvnew.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T14:54:10.983-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T14:54:11.008-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T14:54:11.009-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T14:54:11.009-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas1new.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove kv.couchbase.host 2021-06-09T14:59:02.072-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T14:59:02.072-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T14:59:02.099-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T14:59:02.112-07:00 INFO DockerTestBase [pool-3] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T14:59:02.112-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T14:59:02.113-07:00 INFO DockerTestBase [main] Copying files from docker container 5b659228ba60a: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0011_kv_couchbase_enterprise-6.0.0 2021-06-09T14:59:03.830-07:00 INFO DockerTestBase [main] Stopping docker container: 5b659228ba60a 2021-06-09T14:59:04.197-07:00 INFO DockerTestBase [main] Stopped docker container: 5b659228ba60a 2021-06-09T14:59:04.197-07:00 INFO DockerTestBase [main] Removing docker container: 5b659228ba60a 2021-06-09T14:59:04.208-07:00 INFO DockerTestBase [main] Removed docker container: 5b659228ba60a 2021-06-09T14:59:04.214-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T14:59:04.230-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2085d3f2[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@24ba53bd] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:05.244-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@fd58d91[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@11273c3] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:06.260-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2ab5a1ba[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@24725b44] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:07.274-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1b9e9ac4[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@1559661f] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:08.286-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3746f4d9[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@39810d99] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:09.306-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6364d658[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@d55e5b4] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:10.319-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2b1ebc39[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@25d956f2] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:11.331-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@a2f01b9[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@4e8ff54] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:12.345-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@173afe24[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@52386a07] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:13.357-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1f6bc52b[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$485/0x00000001008d9c40@5a695002] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@1fd4b71e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 44] 2021-06-09T14:59:14.361-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:59:15.365-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:59:16.370-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:59:17.374-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:59:18.378-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:59:19.381-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:59:20.385-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:59:21.406-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T14:59:22.410-07:00 WARN DockerTestBase [pool-3] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T14:59:24.068-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T14:59:24.474-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46001] 2021-06-09T14:59:24.475-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T14:59:25.177-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T14:59:25.177-07:00 INFO DockerTestBase [main] Success 2021-06-09T14:59:25.178-07:00 INFO DockerTestBase [main] Copying files from docker container dcaf57bb602ec: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0014_kvnew_thisbuild 2021-06-09T14:59:25.178-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 76ea0226c5c98: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD/upgradeVerifyClusterCompat/0013_cbas1new_thisbuild 2021-06-09T14:59:26.292-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 76ea0226c5c98 2021-06-09T14:59:26.771-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 76ea0226c5c98 2021-06-09T14:59:27.466-07:00 INFO DockerTestBase [main] Stopping docker container: dcaf57bb602ec 2021-06-09T14:59:27.850-07:00 INFO DockerTestBase [main] Stopped docker container: dcaf57bb602ec 2021-06-09T14:59:27.850-07:00 INFO DockerTestBase [main] Removing docker container: dcaf57bb602ec 2021-06-09T14:59:27.851-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 76ea0226c5c98 2021-06-09T14:59:27.858-07:00 INFO DockerTestBase [main] Removed docker container: dcaf57bb602ec 2021-06-09T14:59:27.862-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 76ea0226c5c98 2021-06-09T14:59:27.862-07:00 INFO DockerTestBase [main] Removing docker network: a6c4d36c14d569e514bca8c0aaa823ebedba7d9ec55aef77e6b048030f1c0613 2021-06-09T14:59:27.923-07:00 INFO DockerTestBase [main] Removed docker network: a6c4d36c14d569e514bca8c0aaa823ebedba7d9ec55aef77e6b048030f1c0613 2021-06-09T14:59:27.924-07:00 WARN TestMethodTracer [main] ### upgradeVerifyClusterCompat SUCCEEDED [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 706.928 s - in com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600OnlineITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD 2021-06-09 14:59:31,939 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T14:59:32.019-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T14:59:32.054-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T14:59:32.055-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T14:59:32.508-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.5.0 2021-06-09T14:59:33.985-07:00 INFO DockerTestBase [main+] >> enterprise-6.5.0: Pulling from library/couchbase 2021-06-09T14:59:34.353-07:00 INFO DockerTestBase [main+] >> 4bbfd2c87b75: Already exists 2021-06-09T14:59:34.354-07:00 INFO DockerTestBase [main+] >> d2e110be24e1: Already exists 2021-06-09T14:59:34.356-07:00 INFO DockerTestBase [main+] >> 889a7173dcfe: Already exists 2021-06-09T14:59:34.358-07:00 INFO DockerTestBase [main+] >> f8723b0ce09e: Already exists 2021-06-09T14:59:34.359-07:00 INFO DockerTestBase [main+] >> 5ad9d59e6d21: Already exists 2021-06-09T14:59:34.362-07:00 INFO DockerTestBase [main+] >> 6c5f58794fc7: Pulling fs layer 2021-06-09T14:59:34.363-07:00 INFO DockerTestBase [main+] >> 38f81fcfdbec: Pulling fs layer 2021-06-09T14:59:34.363-07:00 INFO DockerTestBase [main+] >> 8789570391e0: Pulling fs layer 2021-06-09T14:59:34.364-07:00 INFO DockerTestBase [main+] >> bf58fc1e41e7: Pulling fs layer 2021-06-09T14:59:34.364-07:00 INFO DockerTestBase [main+] >> 7f63430d3051: Pulling fs layer 2021-06-09T14:59:34.364-07:00 INFO DockerTestBase [main+] >> a2029d11a713: Pulling fs layer 2021-06-09T14:59:34.365-07:00 INFO DockerTestBase [main+] >> 4df70b8b6145: Pulling fs layer 2021-06-09T14:59:34.365-07:00 INFO DockerTestBase [main+] >> 93c5ea540823: Pulling fs layer 2021-06-09T14:59:34.365-07:00 INFO DockerTestBase [main+] >> 16e23f3da773: Pulling fs layer 2021-06-09T14:59:34.366-07:00 INFO DockerTestBase [main+] >> a2029d11a713: Waiting 2021-06-09T14:59:34.366-07:00 INFO DockerTestBase [main+] >> 4df70b8b6145: Waiting 2021-06-09T14:59:34.367-07:00 INFO DockerTestBase [main+] >> bf58fc1e41e7: Waiting 2021-06-09T14:59:34.367-07:00 INFO DockerTestBase [main+] >> 7f63430d3051: Waiting 2021-06-09T14:59:34.368-07:00 INFO DockerTestBase [main+] >> 16e23f3da773: Waiting 2021-06-09T14:59:34.368-07:00 INFO DockerTestBase [main+] >> 93c5ea540823: Waiting 2021-06-09T14:59:34.711-07:00 INFO DockerTestBase [main+] >> 8789570391e0: Verifying Checksum 2021-06-09T14:59:34.711-07:00 INFO DockerTestBase [main+] >> 8789570391e0: Download complete 2021-06-09T14:59:34.755-07:00 INFO DockerTestBase [main+] >> 6c5f58794fc7: Verifying Checksum 2021-06-09T14:59:34.756-07:00 INFO DockerTestBase [main+] >> 6c5f58794fc7: Download complete 2021-06-09T14:59:34.916-07:00 INFO DockerTestBase [main+] >> 6c5f58794fc7: Pull complete 2021-06-09T14:59:35.095-07:00 INFO DockerTestBase [main+] >> bf58fc1e41e7: Verifying Checksum 2021-06-09T14:59:35.096-07:00 INFO DockerTestBase [main+] >> bf58fc1e41e7: Download complete 2021-06-09T14:59:35.136-07:00 INFO DockerTestBase [main+] >> 7f63430d3051: Verifying Checksum 2021-06-09T14:59:35.137-07:00 INFO DockerTestBase [main+] >> 7f63430d3051: Download complete 2021-06-09T14:59:35.503-07:00 INFO DockerTestBase [main+] >> a2029d11a713: Verifying Checksum 2021-06-09T14:59:35.503-07:00 INFO DockerTestBase [main+] >> a2029d11a713: Download complete 2021-06-09T14:59:35.509-07:00 INFO DockerTestBase [main+] >> 4df70b8b6145: Verifying Checksum 2021-06-09T14:59:35.509-07:00 INFO DockerTestBase [main+] >> 4df70b8b6145: Download complete 2021-06-09T14:59:35.885-07:00 INFO DockerTestBase [main+] >> 16e23f3da773: Verifying Checksum 2021-06-09T14:59:35.885-07:00 INFO DockerTestBase [main+] >> 16e23f3da773: Download complete 2021-06-09T14:59:35.900-07:00 INFO DockerTestBase [main+] >> 93c5ea540823: Download complete 2021-06-09T14:59:40.310-07:00 INFO DockerTestBase [main+] >> 38f81fcfdbec: Verifying Checksum 2021-06-09T14:59:40.311-07:00 INFO DockerTestBase [main+] >> 38f81fcfdbec: Download complete 2021-06-09T14:59:52.585-07:00 INFO DockerTestBase [main+] >> 38f81fcfdbec: Pull complete 2021-06-09T14:59:53.029-07:00 INFO DockerTestBase [main+] >> 8789570391e0: Pull complete 2021-06-09T14:59:53.390-07:00 INFO DockerTestBase [main+] >> bf58fc1e41e7: Pull complete 2021-06-09T14:59:54.319-07:00 INFO DockerTestBase [main+] >> 7f63430d3051: Pull complete 2021-06-09T14:59:54.605-07:00 INFO DockerTestBase [main+] >> a2029d11a713: Pull complete 2021-06-09T14:59:54.732-07:00 INFO DockerTestBase [main+] >> 4df70b8b6145: Pull complete 2021-06-09T14:59:54.921-07:00 INFO DockerTestBase [main+] >> 93c5ea540823: Pull complete 2021-06-09T14:59:55.051-07:00 INFO DockerTestBase [main+] >> 16e23f3da773: Pull complete 2021-06-09T14:59:55.061-07:00 INFO DockerTestBase [main+] >> Digest: sha256:1b0ced22d7478a9739741d310e3733cbbbbe2bc612453417cba2bcc7df808475 2021-06-09T14:59:55.062-07:00 INFO DockerTestBase [main+] >> Status: Downloaded newer image for couchbase:enterprise-6.5.0 2021-06-09T14:59:55.064-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.5.0 2021-06-09T14:59:55.081-07:00 WARN TestMethodTracer [main] ### upgradeOnlineVerifyClusterCompat START 2021-06-09T14:59:55.133-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T14:59:56.433-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-9cf40373-b7d1-41d1-a05a-3816fd62c945) 2021-06-09T14:59:56.506-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-9cf40373-b7d1-41d1-a05a-3816fd62c945, id = 8ac8c3fca85ccdb8ca2a1e905d87ef61900b504319fc3926d99301a9838af486 } 2021-06-09T14:59:56.508-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.5.0])... 2021-06-09T14:59:56.510-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T14:59:56.512-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0001_n1_couchbase_enterprise-6.5.0, network = { name = cbas-docker-test-0001-9cf40373-b7d1-41d1-a05a-3816fd62c945, id = 8ac8c3fca85ccdb8ca2a1e905d87ef61900b504319fc3926d99301a9838af486 }) 2021-06-09T14:59:56.532-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:34042/tcp 2021-06-09T14:59:56.533-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:34416/tcp 2021-06-09T14:59:56.655-07:00 INFO DockerTestBase [main] Starting docker container: b61bb900966c2 2021-06-09T14:59:58.796-07:00 INFO DockerTestBase [main] Started docker container: b61bb900966c2 2021-06-09T14:59:58.933-07:00 INFO DockerTestBase [main] Container information for b61bb900966c2: host: n1.couchbase.host, image: enterprise-6.5.0, ip: 192.168.176.2, 172.17.0.1:34042->192.168.176.2:8091, 172.17.0.1:34416->192.168.176.2:8095 2021-06-09T15:00:06.086-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T15:00:06.088-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T15:00:06.697-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:00:06.729-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.5.0])... 2021-06-09T15:00:06.730-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T15:00:06.731-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0002_n2_couchbase_enterprise-6.5.0, network = { name = cbas-docker-test-0001-9cf40373-b7d1-41d1-a05a-3816fd62c945, id = 8ac8c3fca85ccdb8ca2a1e905d87ef61900b504319fc3926d99301a9838af486 }) 2021-06-09T15:00:06.731-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:35539/tcp 2021-06-09T15:00:06.732-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:45046/tcp 2021-06-09T15:00:06.788-07:00 INFO DockerTestBase [main] Starting docker container: ab97af62e27c9 2021-06-09T15:00:07.909-07:00 INFO DockerTestBase [main] Started docker container: ab97af62e27c9 2021-06-09T15:00:07.918-07:00 INFO DockerTestBase [main] Container information for ab97af62e27c9: host: n2.couchbase.host, image: enterprise-6.5.0, ip: 192.168.176.3, 172.17.0.1:35539->192.168.176.3:8091, 172.17.0.1:45046->192.168.176.3:8095 2021-06-09T15:00:14.956-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T15:00:14.956-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T15:00:15.813-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:00:15.836-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T15:00:15.837-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T15:00:16.157-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T15:00:16.188-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:00:26.386-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T15:00:26.387-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:00:29.274-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T15:00:29.297-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:00:29.297-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:00:37.840-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:00:37.840-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:00:37.872-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:00:37.881-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:00:37.886-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:00:37.886-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:00:38.079-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:34042] 2021-06-09T15:00:38.085-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:34042 2021-06-09T15:00:39.149-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35539] 2021-06-09T15:00:43.213-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:00:43.216-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:00:43.883-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:00:43.890-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:00:44.893-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:00:45.198-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2903 } 2021-06-09T15:00:45.198-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2903 } 2021-06-09T15:00:46.200-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:00:46.431-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3375 } 2021-06-09T15:00:46.431-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 3375 } 2021-06-09T15:00:47.602-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:00:47.603-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:00:47.604-07:00 INFO UpgradeTestBase [main] Upgrading [n1.couchbase.host] (online, rebalance) ... 2021-06-09T15:00:47.606-07:00 INFO UpgradeTestBase [pool-0] Creating n1-new node using 'this build' docker image... 2021-06-09T15:00:47.607-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n1new to /opt/couchbase/var 2021-06-09T15:00:47.607-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:00:47.735-07:00 INFO DockerTestBase [pool-0+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T15:00:47.735-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T15:00:47.735-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:00:47.737-07:00 INFO DockerTestBase [pool-0+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:00:47.740-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0003_n1new_thisbuild, network = { name = cbas-docker-test-0001-9cf40373-b7d1-41d1-a05a-3816fd62c945, id = 8ac8c3fca85ccdb8ca2a1e905d87ef61900b504319fc3926d99301a9838af486 }) 2021-06-09T15:00:47.741-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8091/tcp -> 172.17.0.1:42368/tcp 2021-06-09T15:00:47.741-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8095/tcp -> 172.17.0.1:43395/tcp 2021-06-09T15:00:47.789-07:00 INFO DockerTestBase [pool-0] Starting docker container: 623c47379dcc4 2021-06-09T15:00:49.177-07:00 INFO DockerTestBase [pool-0] Started docker container: 623c47379dcc4 2021-06-09T15:00:49.186-07:00 INFO DockerTestBase [pool-0] Container information for 623c47379dcc4: host: n1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:42368->192.168.176.4:8091, 172.17.0.1:43395->192.168.176.4:8095 2021-06-09T15:00:59.225-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1new (192.168.176.4) 2021-06-09T15:00:59.225-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n1new.couchbase.host 2021-06-09T15:01:00.009-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T15:01:00.038-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node n1new (192.168.176.4) with services: [DATA, ANALYTICS] 2021-06-09T15:01:00.039-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data,analytics 2021-06-09T15:01:04.560-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T15:01:04.583-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:01:04.583-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:01:04.584-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase --server-remove n1.couchbase.host 2021-06-09T15:04:26.955-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:04:26.955-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:04:27.034-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:04:27.063-07:00 INFO DockerTestBase [pool-2] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:04:27.065-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:04:27.067-07:00 INFO DockerTestBase [main] Copying files from docker container b61bb900966c2: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0001_n1_couchbase_enterprise-6.5.0 2021-06-09T15:04:28.347-07:00 INFO DockerTestBase [main] Stopping docker container: b61bb900966c2 2021-06-09T15:04:28.927-07:00 INFO DockerTestBase [main] Stopped docker container: b61bb900966c2 2021-06-09T15:04:28.928-07:00 INFO DockerTestBase [main] Removing docker container: b61bb900966c2 2021-06-09T15:04:28.944-07:00 INFO DockerTestBase [main] Removed docker container: b61bb900966c2 2021-06-09T15:04:28.947-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:04:28.948-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:04:37.093-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35539] 2021-06-09T15:04:37.093-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42368] 2021-06-09T15:04:37.093-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:04:37.093-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T15:04:37.094-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T15:04:37.094-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:04:37.954-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T15:04:37.954-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T15:04:38.013-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:04:38.013-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:04:38.013-07:00 INFO UpgradeTestBase [main] Upgrading [n2.couchbase.host] (online, rebalance) ... 2021-06-09T15:04:38.014-07:00 INFO UpgradeTestBase [pool-2] Creating n2-new node using 'this build' docker image... 2021-06-09T15:04:38.014-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n2new to /opt/couchbase/var 2021-06-09T15:04:38.015-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0004_n2new_thisbuild, network = { name = cbas-docker-test-0001-9cf40373-b7d1-41d1-a05a-3816fd62c945, id = 8ac8c3fca85ccdb8ca2a1e905d87ef61900b504319fc3926d99301a9838af486 }) 2021-06-09T15:04:38.016-07:00 INFO DockerTestBase [pool-2] binding exposed port n2new:8091/tcp -> 172.17.0.1:42470/tcp 2021-06-09T15:04:38.016-07:00 INFO DockerTestBase [pool-2] binding exposed port n2new:8095/tcp -> 172.17.0.1:44458/tcp 2021-06-09T15:04:38.080-07:00 INFO DockerTestBase [pool-2] Starting docker container: e207b4e35baa1 2021-06-09T15:04:39.326-07:00 INFO DockerTestBase [pool-2] Started docker container: e207b4e35baa1 2021-06-09T15:04:39.343-07:00 INFO DockerTestBase [pool-2] Container information for e207b4e35baa1: host: n2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:42470->192.168.176.2:8091, 172.17.0.1:44458->192.168.176.2:8095 2021-06-09T15:04:50.387-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node n2new (192.168.176.2) 2021-06-09T15:04:50.387-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2new.couchbase.host 2021-06-09T15:04:51.174-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T15:04:51.210-07:00 INFO DockerTestBase [pool-2] Running 'server-add' for node n2new (192.168.176.2) with services: [ANALYTICS] 2021-06-09T15:04:51.210-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://n2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:04:55.498-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Server added 2021-06-09T15:04:55.539-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:04:55.539-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:04:55.540-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove n2.couchbase.host 2021-06-09T15:05:07.155-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:05:07.156-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:05:07.226-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:05:07.270-07:00 INFO DockerTestBase [pool-2] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:05:07.271-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:05:07.271-07:00 INFO DockerTestBase [main] Copying files from docker container ab97af62e27c9: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0002_n2_couchbase_enterprise-6.5.0 2021-06-09T15:05:07.832-07:00 INFO DockerTestBase [main] Stopping docker container: ab97af62e27c9 2021-06-09T15:05:08.186-07:00 INFO DockerTestBase [main] Stopped docker container: ab97af62e27c9 2021-06-09T15:05:08.186-07:00 INFO DockerTestBase [main] Removing docker container: ab97af62e27c9 2021-06-09T15:05:08.197-07:00 INFO DockerTestBase [main] Removed docker container: ab97af62e27c9 2021-06-09T15:05:08.205-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:05:09.936-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:10.939-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:11.942-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:12.945-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:13.949-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:14.952-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:15.956-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:16.959-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:17.962-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:18.966-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:19.969-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:20.973-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:21.976-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:22.979-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:23.983-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:24.987-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:25.990-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:26.994-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:27.998-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:29.001-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:30.008-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:31.018-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:32.029-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:33.035-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:34.038-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:35.042-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:36.046-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:37.049-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:38.052-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:39.056-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:40.060-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:41.063-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:42.066-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:43.069-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:44.073-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:45.077-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:46.080-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:47.083-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:48.087-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:49.091-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:50.094-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:51.097-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:52.100-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:53.105-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:54.109-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:55.113-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:56.116-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:57.119-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:05:58.123-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:05:59.126-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:00.130-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:01.133-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:02.136-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:03.141-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:04.144-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:05.148-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:06.151-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:07.154-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:08.157-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:09.161-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:10.164-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:11.167-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:12.170-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:13.173-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:14.176-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:15.179-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:16.182-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:17.186-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:18.190-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:19.195-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:20.199-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:21.203-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:06:22.207-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:06:23.833-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:06:23.833-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:06:25.141-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42368] 2021-06-09T15:06:25.141-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T15:06:25.141-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42470] 2021-06-09T15:06:25.142-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-32 2021-06-09T15:06:25.142-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:06:26.434-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T15:06:26.434-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T15:06:26.467-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:06:26.468-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:06:26.468-07:00 INFO UpgradeTestBase [main] Verifying metadata 2021-06-09T15:06:26.468-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42368] 2021-06-09T15:06:26.469-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:06:27.043-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 623c47379dcc4: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0003_n1new_thisbuild 2021-06-09T15:06:27.069-07:00 INFO DockerTestBase [main] Copying files from docker container e207b4e35baa1: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD/upgradeOnlineVerifyClusterCompat/0004_n2new_thisbuild 2021-06-09T15:06:27.738-07:00 INFO DockerTestBase [main] Stopping docker container: e207b4e35baa1 2021-06-09T15:06:28.300-07:00 INFO DockerTestBase [main] Stopped docker container: e207b4e35baa1 2021-06-09T15:06:29.235-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 623c47379dcc4 2021-06-09T15:06:29.891-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 623c47379dcc4 2021-06-09T15:06:29.892-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 623c47379dcc4 2021-06-09T15:06:29.892-07:00 INFO DockerTestBase [main] Removing docker container: e207b4e35baa1 2021-06-09T15:06:29.901-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 623c47379dcc4 2021-06-09T15:06:29.903-07:00 INFO DockerTestBase [main] Removed docker container: e207b4e35baa1 2021-06-09T15:06:29.908-07:00 INFO DockerTestBase [main] Removing docker network: 8ac8c3fca85ccdb8ca2a1e905d87ef61900b504319fc3926d99301a9838af486 2021-06-09T15:06:29.973-07:00 INFO DockerTestBase [main] Removed docker network: 8ac8c3fca85ccdb8ca2a1e905d87ef61900b504319fc3926d99301a9838af486 2021-06-09T15:06:29.979-07:00 WARN TestMethodTracer [main] ### upgradeOnlineVerifyClusterCompat SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 420.918 s - in com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom650ITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD 2021-06-09 15:06:33,940 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T15:06:34.021-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T15:06:34.072-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T15:06:34.072-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T15:06:34.278-07:00 WARN TestMethodTracer [main] ### failoverUpgradeAnalytics START 2021-06-09T15:06:34.315-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T15:06:35.857-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d) 2021-06-09T15:06:35.925-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d, id = 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d } 2021-06-09T15:06:35.935-07:00 INFO UpgradeTestBase [pool-0] Creating kv1 node ([couchbase:enterprise-6.5.1])... 2021-06-09T15:06:35.939-07:00 INFO UpgradeTestBase [pool-1] Creating cbas1 node ([couchbase:enterprise-6.5.1])... 2021-06-09T15:06:35.939-07:00 INFO UpgradeTestBase [pool-2] Creating cbas2 node ([couchbase:enterprise-6.5.1])... 2021-06-09T15:06:35.944-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002cbas1 to /opt/couchbase/var 2021-06-09T15:06:35.944-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv1 to /opt/couchbase/var 2021-06-09T15:06:35.945-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003cbas2 to /opt/couchbase/var 2021-06-09T15:06:35.952-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull couchbase:enterprise-6.5.1 2021-06-09T15:06:37.250-07:00 INFO DockerTestBase [pool-0+] >> enterprise-6.5.1: Pulling from library/couchbase 2021-06-09T15:06:37.259-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:e5a07ee624c264d3a8ebcc2ff0989ea710c5843f73b0d541b2e3ea2f476adc13 2021-06-09T15:06:37.259-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for couchbase:enterprise-6.5.1 2021-06-09T15:06:37.260-07:00 INFO DockerTestBase [pool-0+] >> docker.io/library/couchbase:enterprise-6.5.1 2021-06-09T15:06:37.264-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0002_kv1_couchbase_enterprise-6.5.1, network = { name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d, id = 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d }) 2021-06-09T15:06:37.266-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0001_cbas1_couchbase_enterprise-6.5.1, network = { name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d, id = 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d }) 2021-06-09T15:06:37.266-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0003_cbas2_couchbase_enterprise-6.5.1, network = { name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d, id = 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d }) 2021-06-09T15:06:37.299-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8091/tcp -> 172.17.0.1:46779/tcp 2021-06-09T15:06:37.299-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8091/tcp -> 172.17.0.1:39413/tcp 2021-06-09T15:06:37.300-07:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8091/tcp -> 172.17.0.1:46413/tcp 2021-06-09T15:06:37.300-07:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8095/tcp -> 172.17.0.1:37036/tcp 2021-06-09T15:06:37.300-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8095/tcp -> 172.17.0.1:35746/tcp 2021-06-09T15:06:37.300-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8095/tcp -> 172.17.0.1:46660/tcp 2021-06-09T15:06:37.446-07:00 INFO DockerTestBase [pool-2] Starting docker container: 8201a8cd9ef09 2021-06-09T15:06:37.451-07:00 INFO DockerTestBase [pool-1] Starting docker container: ae20229713a26 2021-06-09T15:06:37.451-07:00 INFO DockerTestBase [pool-0] Starting docker container: 57d570356fe3e 2021-06-09T15:06:40.261-07:00 INFO DockerTestBase [pool-1] Started docker container: ae20229713a26 2021-06-09T15:06:40.291-07:00 INFO DockerTestBase [pool-2] Started docker container: 8201a8cd9ef09 2021-06-09T15:06:40.446-07:00 INFO DockerTestBase [pool-1] Container information for ae20229713a26: host: cbas1.couchbase.host, image: enterprise-6.5.1, ip: 192.168.176.3, 172.17.0.1:46413->192.168.176.3:8091, 172.17.0.1:37036->192.168.176.3:8095 2021-06-09T15:06:40.447-07:00 INFO DockerTestBase [pool-2] Container information for 8201a8cd9ef09: host: cbas2.couchbase.host, image: enterprise-6.5.1, ip: 192.168.176.4, 172.17.0.1:39413->192.168.176.4:8091, 172.17.0.1:35746->192.168.176.4:8095 2021-06-09T15:06:41.169-07:00 INFO DockerTestBase [pool-0] Started docker container: 57d570356fe3e 2021-06-09T15:06:41.182-07:00 INFO DockerTestBase [pool-0] Container information for 57d570356fe3e: host: kv1.couchbase.host, image: enterprise-6.5.1, ip: 192.168.176.2, 172.17.0.1:46779->192.168.176.2:8091, 172.17.0.1:46660->192.168.176.2:8095 2021-06-09T15:06:47.851-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node cbas2 (192.168.176.4) 2021-06-09T15:06:47.855-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec cbas2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname cbas2.couchbase.host 2021-06-09T15:06:48.589-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node cbas1 (192.168.176.3) 2021-06-09T15:06:48.590-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cbas1.couchbase.host 2021-06-09T15:06:48.682-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T15:06:49.221-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node kv1 (192.168.176.2) 2021-06-09T15:06:49.221-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv1.couchbase.host 2021-06-09T15:06:49.721-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T15:06:50.335-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T15:06:50.369-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node kv1 (192.168.176.2) with services: [DATA] 2021-06-09T15:06:50.370-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T15:06:50.884-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T15:06:50.913-07:00 INFO DockerTestBase [pool-1] Running 'server-add' for node cbas2 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T15:06:50.914-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node cbas1 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T15:06:50.914-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:06:50.914-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas1.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:06:53.866-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T15:06:56.051-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Server added 2021-06-09T15:06:56.075-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:06:56.076-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:06:57.415-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:06:57.416-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:06:57.486-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:06:57.512-07:00 INFO DockerTestBase [pool-1] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:06:57.513-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:06:57.535-07:00 INFO DockerTestBase [pool-0] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:06:57.634-07:00 INFO KvStoreHttpHelper [pool-1] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46413] 2021-06-09T15:06:57.651-07:00 INFO KvStoreHttpHelper [pool-1] installing sample(s): [beer-sample] on 172.17.0.1:46413 2021-06-09T15:07:07.182-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:46413] 2021-06-09T15:07:09.995-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:07:09.997-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:07:10.467-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:07:10.482-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:07:11.484-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:07:11.930-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1444 } 2021-06-09T15:07:11.930-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 1444 } 2021-06-09T15:07:12.931-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:07:13.046-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:07:13.047-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:07:13.047-07:00 INFO UpgradeFailoverTestBase [main] Failing over (hard) cbas nodes ... 2021-06-09T15:07:13.061-07:00 INFO DockerTestBase [main] Running 'failover --force' on [cbas1 (192.168.176.3), cbas2 (192.168.176.4)] nodes 2021-06-09T15:07:13.062-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli failover -c 192.168.176.2:8091 -u couchbase -p couchbase --server-failover cbas1.couchbase.host,cbas2.couchbase.host --force 2021-06-09T15:07:13.481-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server failed over 2021-06-09T15:07:18.540-07:00 INFO UpgradeTestBase [main] Recreating 'cbas2' node (image = null, forceNewIp = false)... 2021-06-09T15:07:18.540-07:00 INFO DockerTestBase [main] Copying files from docker container 8201a8cd9ef09: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0003_cbas2_couchbase_enterprise-6.5.1 2021-06-09T15:07:18.540-07:00 INFO UpgradeTestBase [ForkJoinPool.commonPool-worker-3] Recreating 'cbas1' node (image = null, forceNewIp = false)... 2021-06-09T15:07:18.542-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container ae20229713a26: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0001_cbas1_couchbase_enterprise-6.5.1 2021-06-09T15:07:18.801-07:00 INFO DockerTestBase [main] Stopping docker container: 8201a8cd9ef09 2021-06-09T15:07:18.820-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: ae20229713a26 2021-06-09T15:07:19.355-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: ae20229713a26 2021-06-09T15:07:19.355-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: ae20229713a26 2021-06-09T15:07:19.366-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: ae20229713a26 2021-06-09T15:07:19.368-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:07:19.402-07:00 INFO DockerTestBase [main] Stopped docker container: 8201a8cd9ef09 2021-06-09T15:07:19.402-07:00 INFO DockerTestBase [main] Removing docker container: 8201a8cd9ef09 2021-06-09T15:07:19.415-07:00 INFO DockerTestBase [main] Removed docker container: 8201a8cd9ef09 2021-06-09T15:07:19.458-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T15:07:19.461-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T15:07:19.461-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:07:19.462-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:07:19.465-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0005_cbas2_thisbuild, network = { name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d, id = 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d }) 2021-06-09T15:07:19.465-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0004_cbas1_thisbuild, network = { name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d, id = 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d }) 2021-06-09T15:07:19.466-07:00 INFO DockerTestBase [main] binding exposed port cbas2:8091/tcp -> 172.17.0.1:38160/tcp 2021-06-09T15:07:19.466-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8091/tcp -> 172.17.0.1:33764/tcp 2021-06-09T15:07:19.466-07:00 INFO DockerTestBase [main] binding exposed port cbas2:8095/tcp -> 172.17.0.1:39670/tcp 2021-06-09T15:07:19.466-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8095/tcp -> 172.17.0.1:42363/tcp 2021-06-09T15:07:19.540-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Starting docker container: 4b850111059ca 2021-06-09T15:07:19.540-07:00 INFO DockerTestBase [main] Starting docker container: 99f34fae78dd6 2021-06-09T15:07:20.688-07:00 INFO DockerTestBase [main] Started docker container: 99f34fae78dd6 2021-06-09T15:07:20.701-07:00 INFO DockerTestBase [main] Container information for 99f34fae78dd6: host: cbas2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:38160->192.168.176.4:8091, 172.17.0.1:39670->192.168.176.4:8095 2021-06-09T15:07:20.763-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Started docker container: 4b850111059ca 2021-06-09T15:07:20.775-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Container information for 4b850111059ca: host: cbas1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:33764->192.168.176.3:8091, 172.17.0.1:42363->192.168.176.3:8095 2021-06-09T15:07:31.741-07:00 INFO DockerTestBase [main] Running full 'recovery' on nodes [cbas1 (192.168.176.3), cbas2 (192.168.176.4)] 2021-06-09T15:07:31.742-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli recovery -c 192.168.176.2:8091 -u couchbase -p couchbase --server-recovery cbas1.couchbase.host,cbas2.couchbase.host --recovery-type full 2021-06-09T15:07:32.136-07:00 INFO DockerTestBase [main+] >> SUCCESS: Servers recovered 2021-06-09T15:07:32.168-07:00 INFO UpgradeTestBase [main] Upgrading [kv1.couchbase.host] (online, rebalance) ... 2021-06-09T15:07:32.170-07:00 INFO UpgradeTestBase [pool-0] Creating kv1-new node using 'this build' docker image... 2021-06-09T15:07:32.171-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004kv1new to /opt/couchbase/var 2021-06-09T15:07:32.171-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0006_kv1new_thisbuild, network = { name = cbas-docker-test-0001-d1989557-2118-4b34-abcf-cd5681a59d1d, id = 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d }) 2021-06-09T15:07:32.172-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1new:8091/tcp -> 172.17.0.1:41871/tcp 2021-06-09T15:07:32.172-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1new:8095/tcp -> 172.17.0.1:43137/tcp 2021-06-09T15:07:32.219-07:00 INFO DockerTestBase [pool-0] Starting docker container: cffcf8dee34b0 2021-06-09T15:07:33.180-07:00 INFO DockerTestBase [pool-0] Started docker container: cffcf8dee34b0 2021-06-09T15:07:33.192-07:00 INFO DockerTestBase [pool-0] Container information for cffcf8dee34b0: host: kv1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.5, 172.17.0.1:41871->192.168.176.5:8091, 172.17.0.1:43137->192.168.176.5:8095 2021-06-09T15:07:43.243-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node kv1new (192.168.176.5) 2021-06-09T15:07:43.243-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname kv1new.couchbase.host 2021-06-09T15:07:44.003-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T15:07:44.026-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node kv1new (192.168.176.5) with services: [DATA] 2021-06-09T15:07:44.026-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://kv1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T15:07:48.258-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T15:07:48.286-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:07:48.287-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:07:48.288-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase --server-remove kv1.couchbase.host 2021-06-09T15:09:35.024-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:09:35.025-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:09:35.105-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:09:35.119-07:00 INFO DockerTestBase [pool-6] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:09:35.121-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:09:35.122-07:00 INFO DockerTestBase [main] Copying files from docker container 57d570356fe3e: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0002_kv1_couchbase_enterprise-6.5.1 2021-06-09T15:09:35.618-07:00 INFO DockerTestBase [main] Stopping docker container: 57d570356fe3e 2021-06-09T15:09:36.030-07:00 INFO DockerTestBase [main] Stopped docker container: 57d570356fe3e 2021-06-09T15:09:36.031-07:00 INFO DockerTestBase [main] Removing docker container: 57d570356fe3e 2021-06-09T15:09:36.039-07:00 INFO DockerTestBase [main] Removed docker container: 57d570356fe3e 2021-06-09T15:09:36.045-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:09:36.045-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:09:36.050-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:36.066-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:37.055-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:37.071-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:38.060-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:38.075-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:39.065-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:39.079-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:40.069-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:40.084-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:41.072-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:41.088-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:42.076-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:42.091-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:43.080-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:43.094-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:44.084-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:44.098-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:45.088-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:45.103-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:46.092-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:46.107-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:47.095-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:47.111-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:48.100-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:48.115-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:49.119-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:09:50.123-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:51.127-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:52.130-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:09:54.108-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:09:54.108-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:09:55.646-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:38160] 2021-06-09T15:09:55.646-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:33764] 2021-06-09T15:09:55.647-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:09:55.647-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T15:09:55.647-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16LE 2021-06-09T15:09:56.761-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 0 } 2021-06-09T15:09:56.762-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:09:56.783-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:09:56.783-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:09:57.762-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T15:09:57.784-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:09:58.668-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 0 } 2021-06-09T15:09:58.669-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 2. ResultSet found : { "$1": 0 } 2021-06-09T15:09:58.680-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:09:58.680-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 0 } 2021-06-09T15:09:59.681-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:10:00.053-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 556 } 2021-06-09T15:10:00.053-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 3. ResultSet found : { "$1": 556 } 2021-06-09T15:10:00.149-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 657 } 2021-06-09T15:10:00.149-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 657 } 2021-06-09T15:10:01.056-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T15:10:01.150-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:10:01.335-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 3019 } 2021-06-09T15:10:01.335-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 4. ResultSet found : { "$1": 3019 } 2021-06-09T15:10:01.497-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3125 } 2021-06-09T15:10:01.497-07:00 INFO DockerTestBase [main] Failed attempt number: 4. ResultSet found : { "$1": 3125 } 2021-06-09T15:10:02.465-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T15:10:02.465-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T15:10:02.498-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:10:02.688-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:10:02.689-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:10:02.691-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 99f34fae78dd6: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0005_cbas2_thisbuild 2021-06-09T15:10:02.696-07:00 INFO DockerTestBase [main] Copying files from docker container 4b850111059ca: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0004_cbas1_thisbuild 2021-06-09T15:10:02.696-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container cffcf8dee34b0: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD/failoverUpgradeAnalytics/0006_kv1new_thisbuild 2021-06-09T15:10:03.378-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 99f34fae78dd6 2021-06-09T15:10:03.869-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: cffcf8dee34b0 2021-06-09T15:10:04.038-07:00 INFO DockerTestBase [main] Stopping docker container: 4b850111059ca 2021-06-09T15:10:04.060-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 99f34fae78dd6 2021-06-09T15:10:04.410-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: cffcf8dee34b0 2021-06-09T15:10:04.500-07:00 INFO DockerTestBase [main] Stopped docker container: 4b850111059ca 2021-06-09T15:10:04.503-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 4b850111059ca 2021-06-09T15:10:04.503-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: cffcf8dee34b0 2021-06-09T15:10:04.506-07:00 INFO DockerTestBase [main] Removing docker container: 99f34fae78dd6 2021-06-09T15:10:04.516-07:00 INFO DockerTestBase [main] Removed docker container: 99f34fae78dd6 2021-06-09T15:10:04.516-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 4b850111059ca 2021-06-09T15:10:04.521-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: cffcf8dee34b0 2021-06-09T15:10:04.523-07:00 INFO DockerTestBase [main] Removing docker network: 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d 2021-06-09T15:10:04.615-07:00 INFO DockerTestBase [main] Removed docker network: 2730a9116ecb22d70a23b021050595d05c7bd38d790baa1d94a4338bdb018c3d 2021-06-09T15:10:04.622-07:00 WARN TestMethodTracer [main] ### failoverUpgradeAnalytics SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 213.665 s - in com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom651FailoverITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD 2021-06-09 15:10:08,274 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T15:10:08.359-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T15:10:08.397-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T15:10:08.397-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T15:10:08.603-07:00 WARN TestMethodTracer [main] ### failoverUpgradeAnalytics START 2021-06-09T15:10:08.642-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T15:10:10.153-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a) 2021-06-09T15:10:10.231-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a, id = edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac } 2021-06-09T15:10:10.237-07:00 INFO UpgradeTestBase [pool-0] Creating kv1 node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:10:10.237-07:00 INFO UpgradeTestBase [pool-2] Creating cbas2 node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:10:10.249-07:00 INFO UpgradeTestBase [pool-1] Creating cbas1 node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:10:10.251-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001cbas2 to /opt/couchbase/var 2021-06-09T15:10:10.251-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002kv1 to /opt/couchbase/var 2021-06-09T15:10:10.255-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003cbas1 to /opt/couchbase/var 2021-06-09T15:10:10.266-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull couchbase:enterprise-6.6.0 2021-06-09T15:10:11.373-07:00 INFO DockerTestBase [pool-0+] >> enterprise-6.6.0: Pulling from library/couchbase 2021-06-09T15:10:11.383-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:bd26854f1bfe7b5cabe25b342356867d58a07d469a8e52f080519eb05818cf81 2021-06-09T15:10:11.384-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for couchbase:enterprise-6.6.0 2021-06-09T15:10:11.386-07:00 INFO DockerTestBase [pool-0+] >> docker.io/library/couchbase:enterprise-6.6.0 2021-06-09T15:10:11.389-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0002_kv1_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a, id = edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac }) 2021-06-09T15:10:11.390-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0001_cbas2_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a, id = edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac }) 2021-06-09T15:10:11.390-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0003_cbas1_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a, id = edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac }) 2021-06-09T15:10:11.428-07:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8091/tcp -> 172.17.0.1:35335/tcp 2021-06-09T15:10:11.428-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8091/tcp -> 172.17.0.1:44336/tcp 2021-06-09T15:10:11.428-07:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8095/tcp -> 172.17.0.1:36708/tcp 2021-06-09T15:10:11.429-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8095/tcp -> 172.17.0.1:41716/tcp 2021-06-09T15:10:11.431-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8091/tcp -> 172.17.0.1:41453/tcp 2021-06-09T15:10:11.435-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8095/tcp -> 172.17.0.1:46256/tcp 2021-06-09T15:10:11.627-07:00 INFO DockerTestBase [pool-0] Starting docker container: 265e909494ade 2021-06-09T15:10:11.625-07:00 INFO DockerTestBase [pool-1] Starting docker container: 1339652c25aec 2021-06-09T15:10:11.625-07:00 INFO DockerTestBase [pool-2] Starting docker container: d773c15c90d19 2021-06-09T15:10:14.423-07:00 INFO DockerTestBase [pool-1] Started docker container: 1339652c25aec 2021-06-09T15:10:14.475-07:00 INFO DockerTestBase [pool-2] Started docker container: d773c15c90d19 2021-06-09T15:10:14.544-07:00 INFO DockerTestBase [pool-0] Started docker container: 265e909494ade 2021-06-09T15:10:14.761-07:00 INFO DockerTestBase [pool-1] Container information for 1339652c25aec: host: cbas1.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.3, 172.17.0.1:35335->192.168.176.3:8091, 172.17.0.1:36708->192.168.176.3:8095 2021-06-09T15:10:14.764-07:00 INFO DockerTestBase [pool-0] Container information for 265e909494ade: host: kv1.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.2, 172.17.0.1:41453->192.168.176.2:8091, 172.17.0.1:46256->192.168.176.2:8095 2021-06-09T15:10:14.763-07:00 INFO DockerTestBase [pool-2] Container information for d773c15c90d19: host: cbas2.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.4, 172.17.0.1:44336->192.168.176.4:8091, 172.17.0.1:41716->192.168.176.4:8095 2021-06-09T15:10:22.238-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node cbas2 (192.168.176.4) 2021-06-09T15:10:22.241-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec cbas2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname cbas2.couchbase.host 2021-06-09T15:10:22.947-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node kv1 (192.168.176.2) 2021-06-09T15:10:22.947-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv1.couchbase.host 2021-06-09T15:10:23.115-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node cbas1 (192.168.176.3) 2021-06-09T15:10:23.115-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cbas1.couchbase.host 2021-06-09T15:10:23.405-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T15:10:24.025-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T15:10:24.186-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T15:10:24.216-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node kv1 (192.168.176.2) with services: [DATA] 2021-06-09T15:10:24.216-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T15:10:24.608-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T15:10:24.642-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node cbas1 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T15:10:24.642-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas1.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:10:24.643-07:00 INFO DockerTestBase [pool-1] Running 'server-add' for node cbas2 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T15:10:24.643-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:10:27.270-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T15:10:29.947-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Server added 2021-06-09T15:10:29.971-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:10:29.972-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:10:31.398-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:10:31.399-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:10:31.450-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:10:31.491-07:00 INFO DockerTestBase [pool-1] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:10:31.493-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:10:31.513-07:00 INFO DockerTestBase [pool-0] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:10:31.621-07:00 INFO KvStoreHttpHelper [pool-1] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35335] 2021-06-09T15:10:31.631-07:00 INFO KvStoreHttpHelper [pool-1] installing sample(s): [beer-sample] on 172.17.0.1:35335 2021-06-09T15:10:43.144-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35335] 2021-06-09T15:10:46.527-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:10:46.530-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:10:47.176-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:10:47.187-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:10:48.193-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:10:48.691-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2193 } 2021-06-09T15:10:48.692-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2193 } 2021-06-09T15:10:49.693-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:10:49.975-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 4228 } 2021-06-09T15:10:49.976-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 4228 } 2021-06-09T15:10:51.102-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:10:51.102-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:10:51.102-07:00 INFO UpgradeFailoverTestBase [main] Failing over (hard) cbas nodes ... 2021-06-09T15:10:51.118-07:00 INFO DockerTestBase [main] Running 'failover --hard' on [cbas1 (192.168.176.3), cbas2 (192.168.176.4)] nodes 2021-06-09T15:10:51.119-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli failover -c 192.168.176.2:8091 -u couchbase -p couchbase --server-failover cbas1.couchbase.host,cbas2.couchbase.host --hard 2021-06-09T15:10:51.530-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server failed over 2021-06-09T15:10:56.619-07:00 INFO UpgradeTestBase [main] Recreating 'cbas2' node (image = null, forceNewIp = false)... 2021-06-09T15:10:56.619-07:00 INFO UpgradeTestBase [ForkJoinPool.commonPool-worker-3] Recreating 'cbas1' node (image = null, forceNewIp = false)... 2021-06-09T15:10:56.620-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 1339652c25aec: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0003_cbas1_couchbase_enterprise-6.6.0 2021-06-09T15:10:56.621-07:00 INFO DockerTestBase [main] Copying files from docker container d773c15c90d19: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0001_cbas2_couchbase_enterprise-6.6.0 2021-06-09T15:10:56.878-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 1339652c25aec 2021-06-09T15:10:56.907-07:00 INFO DockerTestBase [main] Stopping docker container: d773c15c90d19 2021-06-09T15:10:57.393-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 1339652c25aec 2021-06-09T15:10:57.394-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 1339652c25aec 2021-06-09T15:10:57.405-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 1339652c25aec 2021-06-09T15:10:57.407-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:10:57.445-07:00 INFO DockerTestBase [main] Stopped docker container: d773c15c90d19 2021-06-09T15:10:57.445-07:00 INFO DockerTestBase [main] Removing docker container: d773c15c90d19 2021-06-09T15:10:57.456-07:00 INFO DockerTestBase [main] Removed docker container: d773c15c90d19 2021-06-09T15:11:01.538-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T15:11:01.541-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T15:11:01.541-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:11:01.542-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:11:01.545-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0005_cbas2_thisbuild, network = { name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a, id = edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac }) 2021-06-09T15:11:01.545-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0004_cbas1_thisbuild, network = { name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a, id = edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac }) 2021-06-09T15:11:01.546-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8091/tcp -> 172.17.0.1:44093/tcp 2021-06-09T15:11:01.546-07:00 INFO DockerTestBase [main] binding exposed port cbas2:8091/tcp -> 172.17.0.1:42350/tcp 2021-06-09T15:11:01.547-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8095/tcp -> 172.17.0.1:44005/tcp 2021-06-09T15:11:01.547-07:00 INFO DockerTestBase [main] binding exposed port cbas2:8095/tcp -> 172.17.0.1:41270/tcp 2021-06-09T15:11:01.620-07:00 INFO DockerTestBase [main] Starting docker container: 2f4ddf55242fe 2021-06-09T15:11:01.634-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Starting docker container: 836904ffe7e93 2021-06-09T15:11:03.045-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Started docker container: 836904ffe7e93 2021-06-09T15:11:03.054-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Container information for 836904ffe7e93: host: cbas1.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:44093->192.168.176.4:8091, 172.17.0.1:44005->192.168.176.4:8095 2021-06-09T15:11:03.149-07:00 INFO DockerTestBase [main] Started docker container: 2f4ddf55242fe 2021-06-09T15:11:03.171-07:00 INFO DockerTestBase [main] Container information for 2f4ddf55242fe: host: cbas2.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:42350->192.168.176.3:8091, 172.17.0.1:41270->192.168.176.3:8095 2021-06-09T15:11:14.211-07:00 INFO DockerTestBase [main] Running full 'recovery' on nodes [cbas1 (192.168.176.3), cbas2 (192.168.176.4)] 2021-06-09T15:11:14.212-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli recovery -c 192.168.176.2:8091 -u couchbase -p couchbase --server-recovery cbas1.couchbase.host,cbas2.couchbase.host --recovery-type full 2021-06-09T15:11:14.653-07:00 INFO DockerTestBase [main+] >> SUCCESS: Servers recovered 2021-06-09T15:11:14.729-07:00 INFO UpgradeTestBase [main] Upgrading [kv1.couchbase.host] (online, rebalance) ... 2021-06-09T15:11:14.732-07:00 INFO UpgradeTestBase [pool-2] Creating kv1-new node using 'this build' docker image... 2021-06-09T15:11:14.733-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004kv1new to /opt/couchbase/var 2021-06-09T15:11:14.734-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0006_kv1new_thisbuild, network = { name = cbas-docker-test-0001-581e3f40-d4ff-4316-8476-e145233cbf4a, id = edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac }) 2021-06-09T15:11:14.734-07:00 INFO DockerTestBase [pool-2] binding exposed port kv1new:8091/tcp -> 172.17.0.1:44491/tcp 2021-06-09T15:11:14.734-07:00 INFO DockerTestBase [pool-2] binding exposed port kv1new:8095/tcp -> 172.17.0.1:32832/tcp 2021-06-09T15:11:14.787-07:00 INFO DockerTestBase [pool-2] Starting docker container: 42c9494fa2eee 2021-06-09T15:11:16.071-07:00 INFO DockerTestBase [pool-2] Started docker container: 42c9494fa2eee 2021-06-09T15:11:16.078-07:00 INFO DockerTestBase [pool-2] Container information for 42c9494fa2eee: host: kv1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.5, 172.17.0.1:44491->192.168.176.5:8091, 172.17.0.1:32832->192.168.176.5:8095 2021-06-09T15:11:26.123-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node kv1new (192.168.176.5) 2021-06-09T15:11:26.123-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname kv1new.couchbase.host 2021-06-09T15:11:26.926-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T15:11:26.963-07:00 INFO DockerTestBase [pool-2] Running 'server-add' for node kv1new (192.168.176.5) with services: [DATA] 2021-06-09T15:11:26.963-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://kv1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T15:11:31.347-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Server added 2021-06-09T15:11:31.379-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:11:31.380-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:11:31.381-07:00 INFO DockerTestBase [main] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove kv1.couchbase.host 2021-06-09T15:13:15.325-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:13:15.326-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:13:15.464-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:13:15.479-07:00 INFO DockerTestBase [pool-6] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:13:15.481-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:13:15.483-07:00 INFO DockerTestBase [main] Copying files from docker container 265e909494ade: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0002_kv1_couchbase_enterprise-6.6.0 2021-06-09T15:13:16.041-07:00 INFO DockerTestBase [main] Stopping docker container: 265e909494ade 2021-06-09T15:13:16.493-07:00 INFO DockerTestBase [main] Stopped docker container: 265e909494ade 2021-06-09T15:13:16.493-07:00 INFO DockerTestBase [main] Removing docker container: 265e909494ade 2021-06-09T15:13:16.504-07:00 INFO DockerTestBase [main] Removed docker container: 265e909494ade 2021-06-09T15:13:16.511-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:13:16.515-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:16.516-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:13:17.519-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:18.524-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:19.527-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:20.125-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:20.531-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:21.129-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:21.535-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:22.133-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:22.539-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:23.137-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:23.543-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:24.141-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:24.546-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:25.147-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:25.551-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:26.151-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:26.555-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:27.155-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:27.559-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:28.158-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:28.563-07:00 WARN DockerTestBase [pool-6] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:13:29.162-07:00 WARN DockerTestBase [pool-7] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:13:31.059-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:13:31.059-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:13:33.565-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42350] 2021-06-09T15:13:33.566-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:44093] 2021-06-09T15:13:33.566-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:13:33.566-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T15:13:33.567-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:13:33.567-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T15:13:34.457-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 0 } 2021-06-09T15:13:34.458-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:13:34.469-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:13:34.470-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:13:35.459-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-32 2021-06-09T15:13:36.198-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 0 } 2021-06-09T15:13:36.199-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 2. ResultSet found : { "$1": 0 } 2021-06-09T15:13:36.205-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:13:36.205-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 0 } 2021-06-09T15:13:37.199-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16LE 2021-06-09T15:13:37.206-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:13:37.526-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 494 } 2021-06-09T15:13:37.526-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 3. ResultSet found : { "$1": 494 } 2021-06-09T15:13:37.552-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 494 } 2021-06-09T15:13:37.553-07:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 494 } 2021-06-09T15:13:38.527-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-32 2021-06-09T15:13:38.821-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3180 } 2021-06-09T15:13:38.821-07:00 INFO DockerTestBase [main] Failed attempt number: 4. ResultSet found : { "$1": 3180 } 2021-06-09T15:13:38.831-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 3126 } 2021-06-09T15:13:38.832-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 4. ResultSet found : { "$1": 3126 } 2021-06-09T15:13:39.822-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:13:39.833-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16BE 2021-06-09T15:13:39.999-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:13:39.999-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:13:40.038-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T15:13:40.038-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T15:13:40.045-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 836904ffe7e93: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0004_cbas1_thisbuild 2021-06-09T15:13:40.045-07:00 INFO DockerTestBase [main] Copying files from docker container 2f4ddf55242fe: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0005_cbas2_thisbuild 2021-06-09T15:13:40.045-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Copying files from docker container 42c9494fa2eee: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0006_kv1new_thisbuild 2021-06-09T15:13:40.725-07:00 INFO DockerTestBase [main] Stopping docker container: 2f4ddf55242fe 2021-06-09T15:13:41.156-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 836904ffe7e93 2021-06-09T15:13:41.196-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopping docker container: 42c9494fa2eee 2021-06-09T15:13:41.262-07:00 INFO DockerTestBase [main] Stopped docker container: 2f4ddf55242fe 2021-06-09T15:13:41.678-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 836904ffe7e93 2021-06-09T15:13:41.761-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopped docker container: 42c9494fa2eee 2021-06-09T15:13:41.762-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 2f4ddf55242fe 2021-06-09T15:13:41.762-07:00 INFO DockerTestBase [main] Removing docker container: 836904ffe7e93 2021-06-09T15:13:41.763-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 42c9494fa2eee 2021-06-09T15:13:41.773-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 42c9494fa2eee 2021-06-09T15:13:41.774-07:00 INFO DockerTestBase [main] Removed docker container: 836904ffe7e93 2021-06-09T15:13:41.777-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 2f4ddf55242fe 2021-06-09T15:13:41.777-07:00 INFO DockerTestBase [main] Removing docker network: edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac 2021-06-09T15:13:41.832-07:00 INFO DockerTestBase [main] Removed docker network: edb31a33be136aadcdae5db2daf9b3d434d5dc70250873c0776aa4da919899ac 2021-06-09T15:13:41.838-07:00 WARN TestMethodTracer [main] ### failoverUpgradeAnalytics SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 216.251 s - in com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD 2021-06-09 15:13:45,207 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T15:13:45.290-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T15:13:45.327-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T15:13:45.327-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T15:13:45.552-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.6.0 2021-06-09T15:13:46.728-07:00 INFO DockerTestBase [main+] >> enterprise-6.6.0: Pulling from library/couchbase 2021-06-09T15:13:46.734-07:00 INFO DockerTestBase [main+] >> Digest: sha256:bd26854f1bfe7b5cabe25b342356867d58a07d469a8e52f080519eb05818cf81 2021-06-09T15:13:46.734-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for couchbase:enterprise-6.6.0 2021-06-09T15:13:46.737-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.6.0 2021-06-09T15:13:46.744-07:00 WARN TestMethodTracer [main] ### upgradeOnlineVerifyClusterCompat START 2021-06-09T15:13:46.780-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T15:13:48.148-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-91e4e722-d1a7-40fa-90ef-f3ef201b7cd7) 2021-06-09T15:13:48.219-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-91e4e722-d1a7-40fa-90ef-f3ef201b7cd7, id = 502b90eda6bb7c06baac1ad1df13c4524642f65a19f73ad855b11c7b8af98208 } 2021-06-09T15:13:48.220-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:13:48.221-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T15:13:48.224-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0001_n1_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-91e4e722-d1a7-40fa-90ef-f3ef201b7cd7, id = 502b90eda6bb7c06baac1ad1df13c4524642f65a19f73ad855b11c7b8af98208 }) 2021-06-09T15:13:48.242-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:34024/tcp 2021-06-09T15:13:48.243-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:38732/tcp 2021-06-09T15:13:48.355-07:00 INFO DockerTestBase [main] Starting docker container: 6f76462a21b22 2021-06-09T15:13:49.931-07:00 INFO DockerTestBase [main] Started docker container: 6f76462a21b22 2021-06-09T15:13:50.068-07:00 INFO DockerTestBase [main] Container information for 6f76462a21b22: host: n1.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.2, 172.17.0.1:34024->192.168.176.2:8091, 172.17.0.1:38732->192.168.176.2:8095 2021-06-09T15:13:57.184-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T15:13:57.186-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T15:13:57.985-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:13:58.008-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:13:58.008-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T15:13:58.009-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0002_n2_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-91e4e722-d1a7-40fa-90ef-f3ef201b7cd7, id = 502b90eda6bb7c06baac1ad1df13c4524642f65a19f73ad855b11c7b8af98208 }) 2021-06-09T15:13:58.010-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:42659/tcp 2021-06-09T15:13:58.010-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:46182/tcp 2021-06-09T15:13:58.055-07:00 INFO DockerTestBase [main] Starting docker container: 6ea511e2a3668 2021-06-09T15:13:59.586-07:00 INFO DockerTestBase [main] Started docker container: 6ea511e2a3668 2021-06-09T15:13:59.594-07:00 INFO DockerTestBase [main] Container information for 6ea511e2a3668: host: n2.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.3, 172.17.0.1:42659->192.168.176.3:8091, 172.17.0.1:46182->192.168.176.3:8095 2021-06-09T15:14:06.624-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T15:14:06.624-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T15:14:07.383-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:14:07.409-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T15:14:07.411-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T15:14:07.858-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T15:14:07.888-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:14:16.589-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T15:14:16.589-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:14:19.628-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T15:14:19.648-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:14:19.649-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:14:28.001-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:14:28.002-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:14:28.045-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:14:28.052-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:14:28.053-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:14:28.053-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:14:28.148-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:34024] 2021-06-09T15:14:28.152-07:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:34024 2021-06-09T15:14:29.210-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42659] 2021-06-09T15:14:32.774-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:14:32.777-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:14:34.296-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 } 2021-06-09T15:14:34.300-07:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:14:35.301-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:14:35.602-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 3508 } 2021-06-09T15:14:35.602-07:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 3508 } 2021-06-09T15:14:36.603-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:14:36.754-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:14:36.754-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:14:36.756-07:00 INFO UpgradeTestBase [main] Upgrading [n1.couchbase.host] (online, rebalance) ... 2021-06-09T15:14:36.758-07:00 INFO UpgradeTestBase [pool-0] Creating n1-new node using 'this build' docker image... 2021-06-09T15:14:36.759-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n1new to /opt/couchbase/var 2021-06-09T15:14:36.759-07:00 INFO DockerTestBase [pool-0] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:14:36.861-07:00 INFO DockerTestBase [pool-0+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T15:14:36.862-07:00 INFO DockerTestBase [pool-0+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T15:14:36.863-07:00 INFO DockerTestBase [pool-0+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:14:36.867-07:00 INFO DockerTestBase [pool-0+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:14:36.870-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0003_n1new_thisbuild, network = { name = cbas-docker-test-0001-91e4e722-d1a7-40fa-90ef-f3ef201b7cd7, id = 502b90eda6bb7c06baac1ad1df13c4524642f65a19f73ad855b11c7b8af98208 }) 2021-06-09T15:14:36.871-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8091/tcp -> 172.17.0.1:36113/tcp 2021-06-09T15:14:36.871-07:00 INFO DockerTestBase [pool-0] binding exposed port n1new:8095/tcp -> 172.17.0.1:37837/tcp 2021-06-09T15:14:36.923-07:00 INFO DockerTestBase [pool-0] Starting docker container: 792f76269fdcc 2021-06-09T15:14:38.484-07:00 INFO DockerTestBase [pool-0] Started docker container: 792f76269fdcc 2021-06-09T15:14:38.494-07:00 INFO DockerTestBase [pool-0] Container information for 792f76269fdcc: host: n1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:36113->192.168.176.4:8091, 172.17.0.1:37837->192.168.176.4:8095 2021-06-09T15:14:48.535-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node n1new (192.168.176.4) 2021-06-09T15:14:48.536-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n1new.couchbase.host 2021-06-09T15:14:50.223-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T15:14:50.253-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node n1new (192.168.176.4) with services: [DATA, ANALYTICS] 2021-06-09T15:14:50.253-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data,analytics 2021-06-09T15:14:54.457-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T15:14:54.481-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:14:54.481-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:14:54.482-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase --server-remove n1.couchbase.host 2021-06-09T15:18:16.334-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:18:16.334-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:18:16.379-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:18:16.405-07:00 INFO DockerTestBase [pool-2] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:18:16.405-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:18:16.408-07:00 INFO DockerTestBase [main] Copying files from docker container 6f76462a21b22: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0001_n1_couchbase_enterprise-6.6.0 2021-06-09T15:18:17.809-07:00 INFO DockerTestBase [main] Stopping docker container: 6f76462a21b22 2021-06-09T15:18:18.431-07:00 INFO DockerTestBase [main] Stopped docker container: 6f76462a21b22 2021-06-09T15:18:18.432-07:00 INFO DockerTestBase [main] Removing docker container: 6f76462a21b22 2021-06-09T15:18:18.443-07:00 INFO DockerTestBase [main] Removed docker container: 6f76462a21b22 2021-06-09T15:18:18.451-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:18:18.453-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:18:27.658-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42659] 2021-06-09T15:18:27.658-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36113] 2021-06-09T15:18:27.659-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:18:27.659-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T15:18:27.659-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16LE 2021-06-09T15:18:28.382-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T15:18:28.382-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T15:18:28.425-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:18:28.426-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:18:28.426-07:00 INFO UpgradeTestBase [main] Upgrading [n2.couchbase.host] (online, rebalance) ... 2021-06-09T15:18:28.427-07:00 INFO UpgradeTestBase [pool-2] Creating n2-new node using 'this build' docker image... 2021-06-09T15:18:28.427-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n2new to /opt/couchbase/var 2021-06-09T15:18:28.428-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0004_n2new_thisbuild, network = { name = cbas-docker-test-0001-91e4e722-d1a7-40fa-90ef-f3ef201b7cd7, id = 502b90eda6bb7c06baac1ad1df13c4524642f65a19f73ad855b11c7b8af98208 }) 2021-06-09T15:18:28.428-07:00 INFO DockerTestBase [pool-2] binding exposed port n2new:8091/tcp -> 172.17.0.1:42917/tcp 2021-06-09T15:18:28.429-07:00 INFO DockerTestBase [pool-2] binding exposed port n2new:8095/tcp -> 172.17.0.1:32921/tcp 2021-06-09T15:18:28.483-07:00 INFO DockerTestBase [pool-2] Starting docker container: f895f9d05456c 2021-06-09T15:18:30.248-07:00 INFO DockerTestBase [pool-2] Started docker container: f895f9d05456c 2021-06-09T15:18:30.258-07:00 INFO DockerTestBase [pool-2] Container information for f895f9d05456c: host: n2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:42917->192.168.176.2:8091, 172.17.0.1:32921->192.168.176.2:8095 2021-06-09T15:18:40.293-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node n2new (192.168.176.2) 2021-06-09T15:18:40.294-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2new.couchbase.host 2021-06-09T15:18:41.498-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T15:18:41.522-07:00 INFO DockerTestBase [pool-2] Running 'server-add' for node n2new (192.168.176.2) with services: [ANALYTICS] 2021-06-09T15:18:41.523-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://n2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:18:45.677-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Server added 2021-06-09T15:18:45.710-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:18:45.710-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:18:45.711-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove n2.couchbase.host 2021-06-09T15:18:57.366-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:18:57.367-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:18:57.487-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:18:57.496-07:00 INFO DockerTestBase [pool-2] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:18:57.496-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:18:57.498-07:00 INFO DockerTestBase [main] Copying files from docker container 6ea511e2a3668: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0002_n2_couchbase_enterprise-6.6.0 2021-06-09T15:18:58.004-07:00 INFO DockerTestBase [main] Stopping docker container: 6ea511e2a3668 2021-06-09T15:18:58.415-07:00 INFO DockerTestBase [main] Stopped docker container: 6ea511e2a3668 2021-06-09T15:18:58.416-07:00 INFO DockerTestBase [main] Removing docker container: 6ea511e2a3668 2021-06-09T15:18:58.426-07:00 INFO DockerTestBase [main] Removed docker container: 6ea511e2a3668 2021-06-09T15:18:58.429-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:18:58.584-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:19:01.003-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:02.019-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:03.032-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:04.045-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:05.057-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:06.076-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:07.089-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:08.104-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:09.116-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.lang.Exception: HTTP error 503: 2021-06-09T15:19:10.120-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:11.124-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:12.127-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:13.131-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:14.135-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:15.138-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:16.141-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:17.145-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:18.149-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:19.152-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:20.156-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:21.162-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:22.168-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:23.264-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:24.267-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:25.271-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:26.274-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:27.278-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:28.287-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:29.292-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:19:30.296-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:19:31.667-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:19:31.667-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:19:33.119-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42917] 2021-06-09T15:19:33.119-07:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36113] 2021-06-09T15:19:33.119-07:00 INFO DockerTestBase [main] Waiting for all beers to be ingested... 2021-06-09T15:19:33.119-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested... 2021-06-09T15:19:33.120-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:19:33.120-07:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-32 2021-06-09T15:19:34.180-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 5891 } 2021-06-09T15:19:34.181-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Success 2021-06-09T15:19:34.228-07:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 } 2021-06-09T15:19:34.229-07:00 INFO DockerTestBase [main] Success 2021-06-09T15:19:34.229-07:00 INFO UpgradeTestBase [main] Verifying metadata 2021-06-09T15:19:34.230-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36113] 2021-06-09T15:19:34.447-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 792f76269fdcc: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0003_n1new_thisbuild 2021-06-09T15:19:34.447-07:00 INFO DockerTestBase [main] Copying files from docker container f895f9d05456c: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD/upgradeOnlineVerifyClusterCompat/0004_n2new_thisbuild 2021-06-09T15:19:35.081-07:00 INFO DockerTestBase [main] Stopping docker container: f895f9d05456c 2021-06-09T15:19:35.756-07:00 INFO DockerTestBase [main] Stopped docker container: f895f9d05456c 2021-06-09T15:19:36.779-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 792f76269fdcc 2021-06-09T15:19:37.473-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 792f76269fdcc 2021-06-09T15:19:37.474-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 792f76269fdcc 2021-06-09T15:19:37.474-07:00 INFO DockerTestBase [main] Removing docker container: f895f9d05456c 2021-06-09T15:19:37.485-07:00 INFO DockerTestBase [main] Removed docker container: f895f9d05456c 2021-06-09T15:19:37.488-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 792f76269fdcc 2021-06-09T15:19:37.489-07:00 INFO DockerTestBase [main] Removing docker network: 502b90eda6bb7c06baac1ad1df13c4524642f65a19f73ad855b11c7b8af98208 2021-06-09T15:19:37.548-07:00 INFO DockerTestBase [main] Removed docker network: 502b90eda6bb7c06baac1ad1df13c4524642f65a19f73ad855b11c7b8af98208 2021-06-09T15:19:37.554-07:00 WARN TestMethodTracer [main] ### upgradeOnlineVerifyClusterCompat SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 354.734 s - in com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660ITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD 2021-06-09 15:19:40,521 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T15:19:40.600-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T15:19:40.635-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T15:19:40.635-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T15:19:40.812-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.6.2 2021-06-09T15:19:42.194-07:00 INFO DockerTestBase [main+] >> enterprise-6.6.2: Pulling from library/couchbase 2021-06-09T15:19:42.546-07:00 INFO DockerTestBase [main+] >> 345e3491a907: Already exists 2021-06-09T15:19:42.548-07:00 INFO DockerTestBase [main+] >> 57671312ef6f: Already exists 2021-06-09T15:19:42.550-07:00 INFO DockerTestBase [main+] >> 5e9250ddb7d0: Already exists 2021-06-09T15:19:42.553-07:00 INFO DockerTestBase [main+] >> 0e833c94f838: Pulling fs layer 2021-06-09T15:19:42.553-07:00 INFO DockerTestBase [main+] >> e1ec47179f16: Pulling fs layer 2021-06-09T15:19:42.553-07:00 INFO DockerTestBase [main+] >> 24480e43e12c: Pulling fs layer 2021-06-09T15:19:42.553-07:00 INFO DockerTestBase [main+] >> fb49a99c750a: Pulling fs layer 2021-06-09T15:19:42.554-07:00 INFO DockerTestBase [main+] >> 280c455a1a9f: Pulling fs layer 2021-06-09T15:19:42.554-07:00 INFO DockerTestBase [main+] >> 2841994a55ef: Pulling fs layer 2021-06-09T15:19:42.554-07:00 INFO DockerTestBase [main+] >> 678009088519: Pulling fs layer 2021-06-09T15:19:42.554-07:00 INFO DockerTestBase [main+] >> c5776532a164: Pulling fs layer 2021-06-09T15:19:42.554-07:00 INFO DockerTestBase [main+] >> 79a7ba24db9a: Pulling fs layer 2021-06-09T15:19:42.555-07:00 INFO DockerTestBase [main+] >> ea5991001bfb: Pulling fs layer 2021-06-09T15:19:42.555-07:00 INFO DockerTestBase [main+] >> 4c1250c860ad: Pulling fs layer 2021-06-09T15:19:42.555-07:00 INFO DockerTestBase [main+] >> 678009088519: Waiting 2021-06-09T15:19:42.556-07:00 INFO DockerTestBase [main+] >> c5776532a164: Waiting 2021-06-09T15:19:42.556-07:00 INFO DockerTestBase [main+] >> 79a7ba24db9a: Waiting 2021-06-09T15:19:42.556-07:00 INFO DockerTestBase [main+] >> fb49a99c750a: Waiting 2021-06-09T15:19:42.556-07:00 INFO DockerTestBase [main+] >> 280c455a1a9f: Waiting 2021-06-09T15:19:42.556-07:00 INFO DockerTestBase [main+] >> 2841994a55ef: Waiting 2021-06-09T15:19:42.556-07:00 INFO DockerTestBase [main+] >> ea5991001bfb: Waiting 2021-06-09T15:19:42.557-07:00 INFO DockerTestBase [main+] >> 4c1250c860ad: Waiting 2021-06-09T15:19:42.906-07:00 INFO DockerTestBase [main+] >> 24480e43e12c: Verifying Checksum 2021-06-09T15:19:42.906-07:00 INFO DockerTestBase [main+] >> 24480e43e12c: Download complete 2021-06-09T15:19:42.910-07:00 INFO DockerTestBase [main+] >> e1ec47179f16: Download complete 2021-06-09T15:19:43.023-07:00 INFO DockerTestBase [main+] >> 0e833c94f838: Verifying Checksum 2021-06-09T15:19:43.024-07:00 INFO DockerTestBase [main+] >> 0e833c94f838: Download complete 2021-06-09T15:19:43.238-07:00 INFO DockerTestBase [main+] >> 280c455a1a9f: Verifying Checksum 2021-06-09T15:19:43.238-07:00 INFO DockerTestBase [main+] >> 280c455a1a9f: Download complete 2021-06-09T15:19:43.371-07:00 INFO DockerTestBase [main+] >> 2841994a55ef: Verifying Checksum 2021-06-09T15:19:43.372-07:00 INFO DockerTestBase [main+] >> 2841994a55ef: Download complete 2021-06-09T15:19:43.594-07:00 INFO DockerTestBase [main+] >> 678009088519: Verifying Checksum 2021-06-09T15:19:43.595-07:00 INFO DockerTestBase [main+] >> 678009088519: Download complete 2021-06-09T15:19:43.713-07:00 INFO DockerTestBase [main+] >> c5776532a164: Download complete 2021-06-09T15:19:43.872-07:00 INFO DockerTestBase [main+] >> 0e833c94f838: Pull complete 2021-06-09T15:19:43.935-07:00 INFO DockerTestBase [main+] >> 79a7ba24db9a: Verifying Checksum 2021-06-09T15:19:43.936-07:00 INFO DockerTestBase [main+] >> 79a7ba24db9a: Download complete 2021-06-09T15:19:44.024-07:00 INFO DockerTestBase [main+] >> e1ec47179f16: Pull complete 2021-06-09T15:19:44.068-07:00 INFO DockerTestBase [main+] >> ea5991001bfb: Download complete 2021-06-09T15:19:44.176-07:00 INFO DockerTestBase [main+] >> 24480e43e12c: Pull complete 2021-06-09T15:19:44.280-07:00 INFO DockerTestBase [main+] >> 4c1250c860ad: Verifying Checksum 2021-06-09T15:19:44.280-07:00 INFO DockerTestBase [main+] >> 4c1250c860ad: Download complete 2021-06-09T15:19:48.615-07:00 INFO DockerTestBase [main+] >> fb49a99c750a: Verifying Checksum 2021-06-09T15:19:48.615-07:00 INFO DockerTestBase [main+] >> fb49a99c750a: Download complete 2021-06-09T15:20:00.997-07:00 INFO DockerTestBase [main+] >> fb49a99c750a: Pull complete 2021-06-09T15:20:01.315-07:00 INFO DockerTestBase [main+] >> 280c455a1a9f: Pull complete 2021-06-09T15:20:01.913-07:00 INFO DockerTestBase [main+] >> 2841994a55ef: Pull complete 2021-06-09T15:20:02.056-07:00 INFO DockerTestBase [main+] >> 678009088519: Pull complete 2021-06-09T15:20:02.196-07:00 INFO DockerTestBase [main+] >> c5776532a164: Pull complete 2021-06-09T15:20:02.512-07:00 INFO DockerTestBase [main+] >> 79a7ba24db9a: Pull complete 2021-06-09T15:20:02.800-07:00 INFO DockerTestBase [main+] >> ea5991001bfb: Pull complete 2021-06-09T15:20:02.928-07:00 INFO DockerTestBase [main+] >> 4c1250c860ad: Pull complete 2021-06-09T15:20:02.935-07:00 INFO DockerTestBase [main+] >> Digest: sha256:9e09cdee2578a8999a41b77367ff5a719b1d8e7dcb57bbe18ec857f5dda56152 2021-06-09T15:20:02.937-07:00 INFO DockerTestBase [main+] >> Status: Downloaded newer image for couchbase:enterprise-6.6.2 2021-06-09T15:20:02.939-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.6.2 2021-06-09T15:20:02.952-07:00 WARN TestMethodTracer [main] ### runTest START 2021-06-09T15:20:02.994-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T15:20:04.319-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97) 2021-06-09T15:20:04.384-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 } 2021-06-09T15:20:04.390-07:00 INFO UpgradeTestBase [pool-1] Creating kv2 node ([couchbase:enterprise-6.6.2])... 2021-06-09T15:20:04.390-07:00 INFO UpgradeTestBase [pool-0] Creating kv1 node ([couchbase:enterprise-6.6.2])... 2021-06-09T15:20:04.407-07:00 INFO UpgradeTestBase [pool-2] Creating cbas1 node ([couchbase:enterprise-6.6.2])... 2021-06-09T15:20:04.415-07:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002kv1 to /opt/couchbase/var 2021-06-09T15:20:04.415-07:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv2 to /opt/couchbase/var 2021-06-09T15:20:04.418-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003cbas1 to /opt/couchbase/var 2021-06-09T15:20:04.407-07:00 INFO UpgradeTestBase [pool-3] Creating cbas2 node ([couchbase:enterprise-6.6.2])... 2021-06-09T15:20:04.422-07:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0002_kv1_couchbase_enterprise-6.6.2, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:20:04.423-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0003_cbas1_couchbase_enterprise-6.6.2, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:20:04.423-07:00 INFO UpgradeTestBase [pool-3] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004cbas2 to /opt/couchbase/var 2021-06-09T15:20:04.422-07:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0001_kv2_couchbase_enterprise-6.6.2, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:20:04.426-07:00 INFO DockerTestBase [pool-3] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0004_cbas2_couchbase_enterprise-6.6.2, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:20:04.457-07:00 INFO DockerTestBase [pool-3] binding exposed port cbas2:8091/tcp -> 172.17.0.1:33898/tcp 2021-06-09T15:20:04.457-07:00 INFO DockerTestBase [pool-1] binding exposed port kv2:8091/tcp -> 172.17.0.1:45206/tcp 2021-06-09T15:20:04.458-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas1:8091/tcp -> 172.17.0.1:38145/tcp 2021-06-09T15:20:04.459-07:00 INFO DockerTestBase [pool-1] binding exposed port kv2:8095/tcp -> 172.17.0.1:43285/tcp 2021-06-09T15:20:04.459-07:00 INFO DockerTestBase [pool-2] binding exposed port cbas1:8095/tcp -> 172.17.0.1:44738/tcp 2021-06-09T15:20:04.459-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8091/tcp -> 172.17.0.1:36687/tcp 2021-06-09T15:20:04.459-07:00 INFO DockerTestBase [pool-3] binding exposed port cbas2:8095/tcp -> 172.17.0.1:39827/tcp 2021-06-09T15:20:04.460-07:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8095/tcp -> 172.17.0.1:34085/tcp 2021-06-09T15:20:04.618-07:00 INFO DockerTestBase [pool-1] Starting docker container: 8cbefdeb310c1 2021-06-09T15:20:04.619-07:00 INFO DockerTestBase [pool-0] Starting docker container: 804738ba432da 2021-06-09T15:20:04.619-07:00 INFO DockerTestBase [pool-3] Starting docker container: 3c64f9cde454f 2021-06-09T15:20:04.618-07:00 INFO DockerTestBase [pool-2] Starting docker container: 1ba74e272827c 2021-06-09T15:20:07.548-07:00 INFO DockerTestBase [pool-0] Started docker container: 804738ba432da 2021-06-09T15:20:07.624-07:00 INFO DockerTestBase [pool-3] Started docker container: 3c64f9cde454f 2021-06-09T15:20:07.738-07:00 INFO DockerTestBase [pool-3] Container information for 3c64f9cde454f: host: cbas2.couchbase.host, image: enterprise-6.6.2, ip: 192.168.176.4, 172.17.0.1:39827->192.168.176.4:8095, 172.17.0.1:33898->192.168.176.4:8091 2021-06-09T15:20:07.759-07:00 INFO DockerTestBase [pool-0] Container information for 804738ba432da: host: kv1.couchbase.host, image: enterprise-6.6.2, ip: 192.168.176.2, 172.17.0.1:34085->192.168.176.2:8095, 172.17.0.1:36687->192.168.176.2:8091 2021-06-09T15:20:08.511-07:00 INFO DockerTestBase [pool-2] Started docker container: 1ba74e272827c 2021-06-09T15:20:08.520-07:00 INFO DockerTestBase [pool-2] Container information for 1ba74e272827c: host: cbas1.couchbase.host, image: enterprise-6.6.2, ip: 192.168.176.3, 172.17.0.1:44738->192.168.176.3:8095, 172.17.0.1:38145->192.168.176.3:8091 2021-06-09T15:20:08.544-07:00 INFO DockerTestBase [pool-1] Started docker container: 8cbefdeb310c1 2021-06-09T15:20:08.559-07:00 INFO DockerTestBase [pool-1] Container information for 8cbefdeb310c1: host: kv2.couchbase.host, image: enterprise-6.6.2, ip: 192.168.176.5, 172.17.0.1:43285->192.168.176.5:8095, 172.17.0.1:45206->192.168.176.5:8091 2021-06-09T15:20:15.903-07:00 INFO DockerTestBase [pool-0] Running 'node-init' on node kv1 (192.168.176.2) 2021-06-09T15:20:15.908-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv1.couchbase.host 2021-06-09T15:20:16.096-07:00 INFO DockerTestBase [pool-3] Running 'node-init' on node cbas2 (192.168.176.4) 2021-06-09T15:20:16.096-07:00 INFO DockerTestBase [pool-3] Executing the command: docker exec cbas2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname cbas2.couchbase.host 2021-06-09T15:20:16.886-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node cbas1 (192.168.176.3) 2021-06-09T15:20:16.886-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cbas1.couchbase.host 2021-06-09T15:20:17.089-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized 2021-06-09T15:20:17.136-07:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node kv1 (192.168.176.2) with services: [DATA] 2021-06-09T15:20:17.137-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T15:20:17.399-07:00 INFO DockerTestBase [pool-3+] >> SUCCESS: Node initialized 2021-06-09T15:20:17.616-07:00 INFO DockerTestBase [pool-1] Running 'node-init' on node kv2 (192.168.176.5) 2021-06-09T15:20:17.616-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec kv2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname kv2.couchbase.host 2021-06-09T15:20:17.667-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized 2021-06-09T15:20:18.137-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T15:20:18.297-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized 2021-06-09T15:20:18.328-07:00 INFO DockerTestBase [pool-1] Running 'server-add' for node kv2 (192.168.176.5) with services: [DATA] 2021-06-09T15:20:18.328-07:00 INFO DockerTestBase [pool-0] Running 'server-add' for node cbas2 (192.168.176.4) with services: [ANALYTICS] 2021-06-09T15:20:18.329-07:00 INFO DockerTestBase [pool-2] Running 'server-add' for node cbas1 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T15:20:18.329-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas1.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:20:18.330-07:00 INFO DockerTestBase [pool-1] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://kv2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T15:20:18.331-07:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:20:21.214-07:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added 2021-06-09T15:20:23.116-07:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Server added 2021-06-09T15:20:25.527-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Server added 2021-06-09T15:20:25.551-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:20:25.552-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:20:26.958-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:20:26.958-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:20:26.994-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:20:27.012-07:00 INFO DockerTestBase [pool-2] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:20:27.013-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:20:27.028-07:00 INFO DockerTestBase [pool-2] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:20:27.029-07:00 INFO KvStoreHttpHelper [pool-1] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36687, /172.17.0.1:45206] 2021-06-09T15:20:27.052-07:00 INFO KvStoreHttpHelper [pool-1] Bucket (default) does not exist. 2021-06-09T15:20:27.072-07:00 INFO KvStoreHttpHelper [pool-1] Bucket (default) created successfully. 2021-06-09T15:20:27.409-07:00 WARN KvStoreHttpHelper [pool-1] default._default._default upsert doc failed. Retrying after 2s 2021-06-09T15:20:29.435-07:00 WARN KvStoreHttpHelper [pool-1] default._default._default upsert doc failed. Retrying after 2s 2021-06-09T15:20:38.107-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:38145] 2021-06-09T15:20:44.969-07:00 INFO UpgradeMB45902ITD [main] Waiting for all values to be ingested... 2021-06-09T15:20:44.975-07:00 INFO TestExecutor [pool-0] using Accept-Charset: UTF-16BE 2021-06-09T15:20:44.986-07:00 INFO TestExecutor [pool-1] using Accept-Charset: UTF-32 2021-06-09T15:20:44.986-07:00 INFO TestExecutor [pool-2] using Accept-Charset: UTF-16LE 2021-06-09T15:20:45.935-07:00 INFO DockerTestBase [pool-2] ResultSet found : { "$1": 0 } 2021-06-09T15:20:45.935-07:00 INFO DockerTestBase [pool-0] ResultSet found : { "$1": 0 } 2021-06-09T15:20:45.943-07:00 INFO DockerTestBase [pool-2] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:20:45.945-07:00 INFO DockerTestBase [pool-1] ResultSet found : { "$1": 0 } 2021-06-09T15:20:45.946-07:00 INFO DockerTestBase [pool-1] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:20:45.943-07:00 INFO DockerTestBase [pool-0] Failed attempt number: 1. ResultSet found : { "$1": 0 } 2021-06-09T15:20:46.946-07:00 INFO TestExecutor [pool-1] using Accept-Charset: UTF-16LE 2021-06-09T15:20:46.947-07:00 INFO TestExecutor [pool-0] using Accept-Charset: UTF-16BE 2021-06-09T15:20:47.135-07:00 INFO DockerTestBase [pool-1] ResultSet found : { "$1": 10 } 2021-06-09T15:20:47.135-07:00 INFO DockerTestBase [pool-1] Success 2021-06-09T15:20:47.185-07:00 INFO DockerTestBase [pool-0] ResultSet found : { "$1": 10 } 2021-06-09T15:20:47.186-07:00 INFO DockerTestBase [pool-0] Success 2021-06-09T15:20:47.213-07:00 INFO DockerTestBase [pool-2] ResultSet found : { "$1": 10 } 2021-06-09T15:20:47.213-07:00 INFO DockerTestBase [pool-2] Success 2021-06-09T15:20:47.215-07:00 INFO UpgradeTestBase [main] Upgrading [kv1.couchbase.host] (online, rebalance) ... 2021-06-09T15:20:47.216-07:00 INFO UpgradeTestBase [pool-2] Creating kv1-new node using 'this build' docker image... 2021-06-09T15:20:47.217-07:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0005kv1new to /opt/couchbase/var 2021-06-09T15:20:47.217-07:00 INFO DockerTestBase [pool-2] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:20:47.413-07:00 INFO DockerTestBase [pool-2+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T15:20:47.419-07:00 INFO DockerTestBase [pool-2+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T15:20:47.420-07:00 INFO DockerTestBase [pool-2+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:20:47.420-07:00 INFO DockerTestBase [pool-2+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:20:47.427-07:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0005_kv1new_thisbuild, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:20:47.428-07:00 INFO DockerTestBase [pool-2] binding exposed port kv1new:8091/tcp -> 172.17.0.1:36590/tcp 2021-06-09T15:20:47.428-07:00 INFO DockerTestBase [pool-2] binding exposed port kv1new:8095/tcp -> 172.17.0.1:45451/tcp 2021-06-09T15:20:47.479-07:00 INFO DockerTestBase [pool-2] Starting docker container: bc85bafd2d8d6 2021-06-09T15:20:48.452-07:00 INFO DockerTestBase [pool-2] Started docker container: bc85bafd2d8d6 2021-06-09T15:20:48.461-07:00 INFO DockerTestBase [pool-2] Container information for bc85bafd2d8d6: host: kv1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.6, 172.17.0.1:45451->192.168.176.6:8095, 172.17.0.1:36590->192.168.176.6:8091 2021-06-09T15:20:57.854-07:00 INFO DockerTestBase [pool-2] Running 'node-init' on node kv1new (192.168.176.6) 2021-06-09T15:20:57.854-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.6:8091 -u couchbase -p couchbase --node-init-hostname kv1new.couchbase.host 2021-06-09T15:20:58.811-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized 2021-06-09T15:20:58.836-07:00 INFO DockerTestBase [pool-2] Running 'server-add' for node kv1new (192.168.176.6) with services: [DATA] 2021-06-09T15:20:58.836-07:00 INFO DockerTestBase [pool-2] Executing the command: docker exec kv2.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.5:8091 -u couchbase -p couchbase --server-add http://kv1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T15:21:03.283-07:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Server added 2021-06-09T15:21:03.325-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:21:03.326-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:21:03.327-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv2.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.5:8091 -u couchbase -p couchbase --server-remove kv1.couchbase.host 2021-06-09T15:23:08.649-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:23:08.649-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:23:08.679-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:23:08.707-07:00 INFO DockerTestBase [pool-8] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:23:08.707-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:23:08.709-07:00 INFO DockerTestBase [main] Copying files from docker container 804738ba432da: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0002_kv1_couchbase_enterprise-6.6.2 2021-06-09T15:23:09.206-07:00 INFO DockerTestBase [main] Stopping docker container: 804738ba432da 2021-06-09T15:23:09.566-07:00 INFO DockerTestBase [main] Stopped docker container: 804738ba432da 2021-06-09T15:23:09.567-07:00 INFO DockerTestBase [main] Removing docker container: 804738ba432da 2021-06-09T15:23:09.584-07:00 INFO DockerTestBase [main] Removed docker container: 804738ba432da 2021-06-09T15:23:09.585-07:00 INFO UpgradeTestBase [main] Upgrading [kv2.couchbase.host, cbas1.couchbase.host, cbas2.couchbase.host] (online, rebalance) ... 2021-06-09T15:23:09.586-07:00 INFO UpgradeTestBase [pool-8] Creating kv2-new node using 'this build' docker image... 2021-06-09T15:23:09.586-07:00 INFO UpgradeTestBase [pool-9] Creating cbas1-new node using 'this build' docker image... 2021-06-09T15:23:09.586-07:00 INFO UpgradeTestBase [pool-8] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0006kv2new to /opt/couchbase/var 2021-06-09T15:23:09.586-07:00 INFO UpgradeTestBase [pool-10] Creating cbas2-new node using 'this build' docker image... 2021-06-09T15:23:09.586-07:00 INFO UpgradeTestBase [pool-9] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0007cbas1new to /opt/couchbase/var 2021-06-09T15:23:09.587-07:00 INFO UpgradeTestBase [pool-10] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0008cbas2new to /opt/couchbase/var 2021-06-09T15:23:09.587-07:00 INFO DockerTestBase [pool-9] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0007_cbas1new_thisbuild, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:23:09.587-07:00 INFO DockerTestBase [pool-8] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0006_kv2new_thisbuild, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:23:09.587-07:00 INFO DockerTestBase [pool-10] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0008_cbas2new_thisbuild, network = { name = cbas-docker-test-0001-7e5df17e-3a25-46ad-927e-39cc4f659b97, id = 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 }) 2021-06-09T15:23:09.588-07:00 INFO DockerTestBase [pool-8] binding exposed port kv2new:8091/tcp -> 172.17.0.1:35127/tcp 2021-06-09T15:23:09.588-07:00 INFO DockerTestBase [pool-9] binding exposed port cbas1new:8091/tcp -> 172.17.0.1:42204/tcp 2021-06-09T15:23:09.588-07:00 INFO DockerTestBase [pool-8] binding exposed port kv2new:8095/tcp -> 172.17.0.1:32960/tcp 2021-06-09T15:23:09.588-07:00 INFO DockerTestBase [pool-9] binding exposed port cbas1new:8095/tcp -> 172.17.0.1:41838/tcp 2021-06-09T15:23:09.588-07:00 INFO DockerTestBase [pool-10] binding exposed port cbas2new:8091/tcp -> 172.17.0.1:46082/tcp 2021-06-09T15:23:09.588-07:00 INFO DockerTestBase [pool-10] binding exposed port cbas2new:8095/tcp -> 172.17.0.1:41121/tcp 2021-06-09T15:23:09.635-07:00 INFO DockerTestBase [pool-10] Starting docker container: 8da4e56942b54 2021-06-09T15:23:09.638-07:00 INFO DockerTestBase [pool-9] Starting docker container: ecead673b3e62 2021-06-09T15:23:09.702-07:00 INFO DockerTestBase [pool-8] Starting docker container: 309a07d430ef6 2021-06-09T15:23:11.204-07:00 INFO DockerTestBase [pool-10] Started docker container: 8da4e56942b54 2021-06-09T15:23:11.212-07:00 INFO DockerTestBase [pool-10] Container information for 8da4e56942b54: host: cbas2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:41121->192.168.176.2:8095, 172.17.0.1:46082->192.168.176.2:8091 2021-06-09T15:23:11.380-07:00 INFO DockerTestBase [pool-9] Started docker container: ecead673b3e62 2021-06-09T15:23:11.395-07:00 INFO DockerTestBase [pool-9] Container information for ecead673b3e62: host: cbas1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.7, 172.17.0.1:41838->192.168.176.7:8095, 172.17.0.1:42204->192.168.176.7:8091 2021-06-09T15:23:11.502-07:00 INFO DockerTestBase [pool-8] Started docker container: 309a07d430ef6 2021-06-09T15:23:11.514-07:00 INFO DockerTestBase [pool-8] Container information for 309a07d430ef6: host: kv2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.8, 172.17.0.1:32960->192.168.176.8:8095, 172.17.0.1:35127->192.168.176.8:8091 2021-06-09T15:23:22.274-07:00 INFO DockerTestBase [pool-10] Running 'node-init' on node cbas2new (192.168.176.2) 2021-06-09T15:23:22.275-07:00 INFO DockerTestBase [pool-10] Executing the command: docker exec cbas2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname cbas2new.couchbase.host 2021-06-09T15:23:22.557-07:00 INFO DockerTestBase [pool-8] Running 'node-init' on node kv2new (192.168.176.8) 2021-06-09T15:23:22.557-07:00 INFO DockerTestBase [pool-8] Executing the command: docker exec kv2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.8:8091 -u couchbase -p couchbase --node-init-hostname kv2new.couchbase.host 2021-06-09T15:23:23.453-07:00 INFO DockerTestBase [pool-9] Running 'node-init' on node cbas1new (192.168.176.7) 2021-06-09T15:23:23.454-07:00 INFO DockerTestBase [pool-9] Executing the command: docker exec cbas1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.7:8091 -u couchbase -p couchbase --node-init-hostname cbas1new.couchbase.host 2021-06-09T15:23:24.130-07:00 INFO DockerTestBase [pool-10+] >> SUCCESS: Node initialized 2021-06-09T15:23:24.169-07:00 INFO DockerTestBase [pool-10] Running 'server-add' for node cbas2new (192.168.176.2) with services: [ANALYTICS] 2021-06-09T15:23:24.170-07:00 INFO DockerTestBase [pool-10] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.6:8091 -u couchbase -p couchbase --server-add http://cbas2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:23:24.300-07:00 INFO DockerTestBase [pool-8+] >> SUCCESS: Node initialized 2021-06-09T15:23:24.340-07:00 INFO DockerTestBase [pool-8] Running 'server-add' for node kv2new (192.168.176.8) with services: [DATA] 2021-06-09T15:23:24.341-07:00 INFO DockerTestBase [pool-8] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.6:8091 -u couchbase -p couchbase --server-add http://kv2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T15:23:24.384-07:00 INFO DockerTestBase [pool-9+] >> SUCCESS: Node initialized 2021-06-09T15:23:24.421-07:00 INFO DockerTestBase [pool-9] Running 'server-add' for node cbas1new (192.168.176.7) with services: [ANALYTICS] 2021-06-09T15:23:24.422-07:00 INFO DockerTestBase [pool-9] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.6:8091 -u couchbase -p couchbase --server-add http://cbas1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:23:28.796-07:00 INFO DockerTestBase [pool-10+] >> SUCCESS: Server added 2021-06-09T15:23:32.828-07:00 INFO DockerTestBase [pool-8+] >> SUCCESS: Server added 2021-06-09T15:23:38.086-07:00 INFO DockerTestBase [pool-9+] >> SUCCESS: Server added 2021-06-09T15:23:38.115-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:23:38.115-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:23:38.116-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.6:8091 -u couchbase -p couchbase --server-remove kv2.couchbase.host,cbas1.couchbase.host,cbas2.couchbase.host 2021-06-09T15:26:15.157-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:26:15.157-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:26:15.205-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:26:15.215-07:00 INFO DockerTestBase [pool-14] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:26:36.552-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:26:36.555-07:00 INFO DockerTestBase [main] Copying files from docker container 1ba74e272827c: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0003_cbas1_couchbase_enterprise-6.6.2 2021-06-09T15:26:36.567-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 8cbefdeb310c1: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0001_kv2_couchbase_enterprise-6.6.2 2021-06-09T15:26:36.567-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Copying files from docker container 3c64f9cde454f: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0004_cbas2_couchbase_enterprise-6.6.2 2021-06-09T15:26:37.409-07:00 INFO DockerTestBase [main] Stopping docker container: 1ba74e272827c 2021-06-09T15:26:37.504-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopping docker container: 3c64f9cde454f 2021-06-09T15:26:37.803-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 8cbefdeb310c1 2021-06-09T15:26:38.107-07:00 INFO DockerTestBase [main] Stopped docker container: 1ba74e272827c 2021-06-09T15:26:38.108-07:00 INFO DockerTestBase [main] Removing docker container: 1ba74e272827c 2021-06-09T15:26:38.118-07:00 INFO DockerTestBase [main] Removed docker container: 1ba74e272827c 2021-06-09T15:26:38.199-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopped docker container: 3c64f9cde454f 2021-06-09T15:26:38.200-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 3c64f9cde454f 2021-06-09T15:26:38.227-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 3c64f9cde454f 2021-06-09T15:26:38.387-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 8cbefdeb310c1 2021-06-09T15:26:38.387-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 8cbefdeb310c1 2021-06-09T15:26:38.394-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 8cbefdeb310c1 2021-06-09T15:26:38.402-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:26:38.821-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:26:38.936-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42204] 2021-06-09T15:26:38.938-07:00 INFO TestExecutor [pool-15] using Accept-Charset: UTF-32 2021-06-09T15:26:38.940-07:00 INFO TestExecutor [pool-14] using Accept-Charset: UTF-16LE 2021-06-09T15:26:40.140-07:00 INFO DockerTestBase [pool-15] ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.140-07:00 INFO DockerTestBase [pool-15] Success 2021-06-09T15:26:40.147-07:00 INFO DockerTestBase [pool-16] ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.147-07:00 INFO DockerTestBase [pool-16] Success 2021-06-09T15:26:40.151-07:00 INFO DockerTestBase [pool-14] ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.151-07:00 INFO DockerTestBase [pool-14] Success 2021-06-09T15:26:40.152-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:36590, /172.17.0.1:35127] 2021-06-09T15:26:40.260-07:00 INFO UpgradeMB45902ITD [main] inserted 10 values after upgrade 2021-06-09T15:26:40.261-07:00 INFO TestExecutor [pool-16] using Accept-Charset: UTF-16BE 2021-06-09T15:26:40.261-07:00 INFO TestExecutor [pool-14] using Accept-Charset: UTF-32 2021-06-09T15:26:40.619-07:00 INFO DockerTestBase [pool-16] ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.620-07:00 INFO DockerTestBase [pool-16] Failed attempt number: 1. ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.624-07:00 INFO DockerTestBase [pool-15] ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.624-07:00 INFO DockerTestBase [pool-15] Failed attempt number: 1. ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.712-07:00 INFO DockerTestBase [pool-14] ResultSet found : { "$1": 10 } 2021-06-09T15:26:40.712-07:00 INFO DockerTestBase [pool-14] Failed attempt number: 1. ResultSet found : { "$1": 10 } 2021-06-09T15:26:41.621-07:00 INFO TestExecutor [pool-16] using Accept-Charset: UTF-16LE 2021-06-09T15:26:41.625-07:00 INFO TestExecutor [pool-15] using Accept-Charset: UTF-16BE 2021-06-09T15:26:41.713-07:00 INFO TestExecutor [pool-14] using Accept-Charset: UTF-32 2021-06-09T15:26:41.785-07:00 INFO DockerTestBase [pool-15] ResultSet found : { "$1": 20 } 2021-06-09T15:26:41.785-07:00 INFO DockerTestBase [pool-15] Success 2021-06-09T15:26:41.786-07:00 INFO DockerTestBase [pool-16] ResultSet found : { "$1": 20 } 2021-06-09T15:26:41.786-07:00 INFO DockerTestBase [pool-16] Success 2021-06-09T15:26:41.825-07:00 INFO DockerTestBase [pool-14] ResultSet found : { "$1": 20 } 2021-06-09T15:26:41.825-07:00 INFO DockerTestBase [pool-14] Success 2021-06-09T15:26:41.936-07:00 INFO UpgradeMB45902ITD [main] deleted all values from before upgrade 2021-06-09T15:26:41.937-07:00 INFO TestExecutor [pool-14] using Accept-Charset: UTF-16LE 2021-06-09T15:26:41.938-07:00 INFO TestExecutor [pool-15] using Accept-Charset: UTF-16BE 2021-06-09T15:26:42.101-07:00 INFO DockerTestBase [pool-14] ResultSet found : { "$1": 10 } 2021-06-09T15:26:42.101-07:00 INFO DockerTestBase [pool-14] Success 2021-06-09T15:26:42.110-07:00 INFO DockerTestBase [pool-15] ResultSet found : { "$1": 10 } 2021-06-09T15:26:42.111-07:00 INFO DockerTestBase [pool-15] Success 2021-06-09T15:26:42.112-07:00 INFO DockerTestBase [pool-16] ResultSet found : { "$1": 10 } 2021-06-09T15:26:42.112-07:00 INFO DockerTestBase [pool-16] Success 2021-06-09T15:26:42.161-07:00 INFO KvStoreHttpHelper [main] Request for creation of scope 's1' on bucket 'default' posted successfully. 2021-06-09T15:26:42.285-07:00 INFO KvStoreHttpHelper [main] Request for creation of scope 's2' on bucket 'default' posted successfully. 2021-06-09T15:26:42.422-07:00 INFO KvStoreHttpHelper [main] request for creation of collection (c1) posted successfully. 2021-06-09T15:26:42.517-07:00 INFO KvStoreHttpHelper [main] request for creation of collection (c2) posted successfully. 2021-06-09T15:26:42.538-07:00 WARN KvStoreHttpHelper [Thread-22] default.s1.c1 upsert doc failed. Retrying after 2s 2021-06-09T15:26:42.617-07:00 WARN KvStoreHttpHelper [Thread-21] default.s2.c2 upsert doc failed. Retrying after 2s 2021-06-09T15:26:47.532-07:00 INFO KvUpserter [Thread-22] KvUpserter stopped. Successful upserts: 410, Failed upserts: 0 2021-06-09T15:26:47.533-07:00 INFO KvUpserter [Thread-21] KvUpserter stopped. Successful upserts: 410, Failed upserts: 0 2021-06-09T15:26:55.765-07:00 INFO TestExecutor [pool-16] using Accept-Charset: UTF-32 2021-06-09T15:26:56.042-07:00 INFO DockerTestBase [pool-15] ResultSet found : { "$1": 410 } 2021-06-09T15:26:56.043-07:00 INFO DockerTestBase [pool-15] Success 2021-06-09T15:26:56.049-07:00 INFO DockerTestBase [pool-16] ResultSet found : { "$1": 410 } 2021-06-09T15:26:56.050-07:00 INFO DockerTestBase [pool-16] Success 2021-06-09T15:26:56.051-07:00 INFO TestExecutor [pool-15] using Accept-Charset: UTF-16LE 2021-06-09T15:26:56.051-07:00 INFO TestExecutor [pool-14] using Accept-Charset: UTF-16BE 2021-06-09T15:26:56.053-07:00 INFO TestExecutor [pool-16] using Accept-Charset: UTF-32 2021-06-09T15:26:56.327-07:00 INFO DockerTestBase [pool-16] ResultSet found : { "$1": 10 } 2021-06-09T15:26:56.327-07:00 INFO DockerTestBase [pool-16] Success 2021-06-09T15:26:56.359-07:00 INFO DockerTestBase [pool-15] ResultSet found : { "$1": 10 } 2021-06-09T15:26:56.360-07:00 INFO DockerTestBase [pool-15] Success 2021-06-09T15:26:56.381-07:00 INFO DockerTestBase [pool-14] ResultSet found : { "$1": 10 } 2021-06-09T15:26:56.381-07:00 INFO DockerTestBase [pool-14] Success 2021-06-09T15:26:56.382-07:00 INFO DockerTestBase [main] Copying files from docker container ecead673b3e62: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0007_cbas1new_thisbuild 2021-06-09T15:26:56.382-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Copying files from docker container 309a07d430ef6: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0006_kv2new_thisbuild 2021-06-09T15:26:56.382-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 8da4e56942b54: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0008_cbas2new_thisbuild 2021-06-09T15:26:56.384-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Copying files from docker container bc85bafd2d8d6: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD/runTest/0005_kv1new_thisbuild 2021-06-09T15:26:57.970-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 8da4e56942b54 2021-06-09T15:26:58.275-07:00 INFO DockerTestBase [main] Stopping docker container: ecead673b3e62 2021-06-09T15:26:58.487-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopping docker container: 309a07d430ef6 2021-06-09T15:26:59.424-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 8da4e56942b54 2021-06-09T15:26:59.494-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Stopped docker container: 309a07d430ef6 2021-06-09T15:26:59.536-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopping docker container: bc85bafd2d8d6 2021-06-09T15:26:59.562-07:00 INFO DockerTestBase [main] Stopped docker container: ecead673b3e62 2021-06-09T15:27:00.005-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopped docker container: bc85bafd2d8d6 2021-06-09T15:27:00.006-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 8da4e56942b54 2021-06-09T15:27:00.006-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: bc85bafd2d8d6 2021-06-09T15:27:00.006-07:00 INFO DockerTestBase [main] Removing docker container: ecead673b3e62 2021-06-09T15:27:00.008-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 309a07d430ef6 2021-06-09T15:27:00.017-07:00 INFO DockerTestBase [main] Removed docker container: ecead673b3e62 2021-06-09T15:27:00.018-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 309a07d430ef6 2021-06-09T15:27:00.021-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 8da4e56942b54 2021-06-09T15:27:00.022-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: bc85bafd2d8d6 2021-06-09T15:27:00.023-07:00 INFO DockerTestBase [main] Removing docker network: 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 2021-06-09T15:27:00.076-07:00 INFO DockerTestBase [main] Removed docker network: 4cae02e7bb4303240c77be231b3b9b4d403d363867fd71025041036ecd967899 2021-06-09T15:27:00.086-07:00 WARN TestMethodTracer [main] ### runTest SUCCEEDED [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 441.655 s - in com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeMB45902ITD [INFO] Running com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD 2021-06-09 15:27:04,730 main INFO Registered Log4j as the java.util.logging.LogManager. 2021-06-09T15:27:04.808-07:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork 2021-06-09T15:27:04.843-07:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1 2021-06-09T15:27:04.844-07:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 dockerHostAddr=172.17.0.1 dockerContainerId=aa57276ac8053424c18e17ba982a429baf3794728a715cef3fa737c4972c51d8 2021-06-09T15:27:05.165-07:00 INFO DockerTestBase [main] Executing the command: docker pull couchbase:enterprise-6.6.0 2021-06-09T15:27:06.226-07:00 INFO DockerTestBase [main+] >> enterprise-6.6.0: Pulling from library/couchbase 2021-06-09T15:27:06.234-07:00 INFO DockerTestBase [main+] >> Digest: sha256:bd26854f1bfe7b5cabe25b342356867d58a07d469a8e52f080519eb05818cf81 2021-06-09T15:27:06.234-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for couchbase:enterprise-6.6.0 2021-06-09T15:27:06.235-07:00 INFO DockerTestBase [main+] >> docker.io/library/couchbase:enterprise-6.6.0 2021-06-09T15:27:06.245-07:00 WARN TestMethodTracer [main] ### upgradeOfflineAndDowngrade START 2021-06-09T15:27:06.280-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T15:27:07.688-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9) 2021-06-09T15:27:07.756-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9, id = 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc } 2021-06-09T15:27:07.758-07:00 INFO UpgradeTestBase [main] Creating kv node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:27:07.760-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv to /opt/couchbase/var 2021-06-09T15:27:07.763-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0001_kv_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9, id = 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc }) 2021-06-09T15:27:07.783-07:00 INFO DockerTestBase [main] binding exposed port kv:8091/tcp -> 172.17.0.1:45737/tcp 2021-06-09T15:27:07.784-07:00 INFO DockerTestBase [main] binding exposed port kv:8095/tcp -> 172.17.0.1:37672/tcp 2021-06-09T15:27:07.918-07:00 INFO DockerTestBase [main] Starting docker container: 62dc8b190510f 2021-06-09T15:27:09.900-07:00 INFO DockerTestBase [main] Started docker container: 62dc8b190510f 2021-06-09T15:27:10.030-07:00 INFO DockerTestBase [main] Container information for 62dc8b190510f: host: kv.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.2, 172.17.0.1:45737->192.168.176.2:8091, 172.17.0.1:37672->192.168.176.2:8095 2021-06-09T15:27:17.175-07:00 INFO DockerTestBase [main] Running 'node-init' on node kv (192.168.176.2) 2021-06-09T15:27:17.177-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv.couchbase.host 2021-06-09T15:27:18.395-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:27:18.420-07:00 INFO UpgradeTestBase [main] Creating cc node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:27:18.421-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002cc to /opt/couchbase/var 2021-06-09T15:27:18.422-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0002_cc_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9, id = 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc }) 2021-06-09T15:27:18.422-07:00 INFO DockerTestBase [main] binding exposed port cc:8091/tcp -> 172.17.0.1:42058/tcp 2021-06-09T15:27:18.422-07:00 INFO DockerTestBase [main] binding exposed port cc:8095/tcp -> 172.17.0.1:43484/tcp 2021-06-09T15:27:18.471-07:00 INFO DockerTestBase [main] Starting docker container: beaabd7996400 2021-06-09T15:27:19.769-07:00 INFO DockerTestBase [main] Started docker container: beaabd7996400 2021-06-09T15:27:19.778-07:00 INFO DockerTestBase [main] Container information for beaabd7996400: host: cc.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.3, 172.17.0.1:42058->192.168.176.3:8091, 172.17.0.1:43484->192.168.176.3:8095 2021-06-09T15:27:25.807-07:00 INFO DockerTestBase [main] Running 'node-init' on node cc (192.168.176.3) 2021-06-09T15:27:25.808-07:00 INFO DockerTestBase [main] Executing the command: docker exec cc.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cc.couchbase.host 2021-06-09T15:27:26.539-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:27:26.559-07:00 INFO UpgradeTestBase [main] Creating nc node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:27:26.560-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003nc to /opt/couchbase/var 2021-06-09T15:27:26.561-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0003_nc_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9, id = 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc }) 2021-06-09T15:27:26.561-07:00 INFO DockerTestBase [main] binding exposed port nc:8091/tcp -> 172.17.0.1:35804/tcp 2021-06-09T15:27:26.562-07:00 INFO DockerTestBase [main] binding exposed port nc:8095/tcp -> 172.17.0.1:44409/tcp 2021-06-09T15:27:26.607-07:00 INFO DockerTestBase [main] Starting docker container: f7d9fde8e8fbe 2021-06-09T15:27:28.035-07:00 INFO DockerTestBase [main] Started docker container: f7d9fde8e8fbe 2021-06-09T15:27:28.043-07:00 INFO DockerTestBase [main] Container information for f7d9fde8e8fbe: host: nc.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.4, 172.17.0.1:35804->192.168.176.4:8091, 172.17.0.1:44409->192.168.176.4:8095 2021-06-09T15:27:35.071-07:00 INFO DockerTestBase [main] Running 'node-init' on node nc (192.168.176.4) 2021-06-09T15:27:35.072-07:00 INFO DockerTestBase [main] Executing the command: docker exec nc.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname nc.couchbase.host 2021-06-09T15:27:35.705-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:27:35.730-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node kv (192.168.176.2) with services: [DATA] 2021-06-09T15:27:35.731-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data 2021-06-09T15:27:36.061-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T15:27:36.082-07:00 INFO DockerTestBase [main] Running 'server-add' for node cc (192.168.176.3) with services: [ANALYTICS] 2021-06-09T15:27:36.083-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cc.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:27:38.946-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T15:27:38.965-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:27:38.965-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:27:40.359-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:27:40.360-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:27:40.396-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:27:40.426-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:27:40.427-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:27:40.427-07:00 INFO DockerTestBase [main] Running 'server-add' for node nc (192.168.176.4) with services: [ANALYTICS] 2021-06-09T15:27:40.428-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://nc.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:27:43.674-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T15:27:43.693-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:27:43.694-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:27:51.096-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:27:51.096-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:27:51.117-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:27:51.137-07:00 INFO DockerTestBase [pool-0] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:27:51.138-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:27:51.148-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:27:51.676-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42058] 2021-06-09T15:27:51.677-07:00 INFO SecondaryCompositeIndexITD [main] creating bucket1 and bucket2... 2021-06-09T15:27:51.688-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket1) does not exist. 2021-06-09T15:27:51.699-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket1) created successfully. 2021-06-09T15:27:51.786-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket2) does not exist. 2021-06-09T15:27:51.825-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket2) created successfully. 2021-06-09T15:27:51.868-07:00 INFO SecondaryCompositeIndexITD [main] creating datasets and indexes... 2021-06-09T15:27:53.221-07:00 INFO UpgradeTestBase [main] Connecting link... 2021-06-09T15:27:56.523-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with values, nulls and missings... 2021-06-09T15:27:56.900-07:00 INFO UpgradeTestBase [main] Creating kv-new node using 'this build' docker image... 2021-06-09T15:27:56.901-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004kvnew to /opt/couchbase/var 2021-06-09T15:27:56.902-07:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:27:57.008-07:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.1-1541690320: Pulling from couchbase/server-toy 2021-06-09T15:27:57.009-07:00 INFO DockerTestBase [main+] >> Digest: sha256:a328f472e09f4e50237752e95eeca5444b4913e3e42c8f084718987f664123e3 2021-06-09T15:27:57.010-07:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:27:57.010-07:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.1-1541690320 2021-06-09T15:27:57.014-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0004_kvnew_thisbuild, network = { name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9, id = 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc }) 2021-06-09T15:27:57.015-07:00 INFO DockerTestBase [main] binding exposed port kvnew:8091/tcp -> 172.17.0.1:45721/tcp 2021-06-09T15:27:57.015-07:00 INFO DockerTestBase [main] binding exposed port kvnew:8095/tcp -> 172.17.0.1:44689/tcp 2021-06-09T15:27:57.067-07:00 INFO DockerTestBase [main] Starting docker container: 0173b0d0bdd20 2021-06-09T15:27:58.217-07:00 INFO DockerTestBase [main] Started docker container: 0173b0d0bdd20 2021-06-09T15:27:58.234-07:00 INFO DockerTestBase [main] Container information for 0173b0d0bdd20: host: kvnew.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.5, 172.17.0.1:45721->192.168.176.5:8091, 172.17.0.1:44689->192.168.176.5:8095 2021-06-09T15:28:09.279-07:00 INFO DockerTestBase [main] Running 'node-init' on node kvnew (192.168.176.5) 2021-06-09T15:28:09.280-07:00 INFO DockerTestBase [main] Executing the command: docker exec kvnew.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname kvnew.couchbase.host 2021-06-09T15:28:10.373-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:28:10.419-07:00 INFO DockerTestBase [main] Running 'server-add' for node kvnew (192.168.176.5) with services: [DATA] 2021-06-09T15:28:10.420-07:00 INFO DockerTestBase [main] Executing the command: docker exec kv.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://kvnew.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data 2021-06-09T15:28:14.726-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T15:28:14.746-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:28:14.747-07:00 INFO DockerTestBase [main] Executing the command: docker exec kvnew.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.5:8091 -u couchbase -p couchbase --server-remove kv.couchbase.host 2021-06-09T15:31:56.275-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:31:56.275-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:31:56.307-07:00 INFO UpgradeTestBase [main] Recreating 'cc' node (image = null, forceNewIp = false)... 2021-06-09T15:31:56.308-07:00 INFO DockerTestBase [main] Copying files from docker container beaabd7996400: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0002_cc_couchbase_enterprise-6.6.0 2021-06-09T15:31:56.798-07:00 INFO DockerTestBase [main] Stopping docker container: beaabd7996400 2021-06-09T15:31:57.330-07:00 INFO DockerTestBase [main] Stopped docker container: beaabd7996400 2021-06-09T15:31:57.330-07:00 INFO DockerTestBase [main] Removing docker container: beaabd7996400 2021-06-09T15:31:57.343-07:00 INFO DockerTestBase [main] Removed docker container: beaabd7996400 2021-06-09T15:31:57.345-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0005_cc_thisbuild, network = { name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9, id = 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc }) 2021-06-09T15:31:57.346-07:00 INFO DockerTestBase [main] binding exposed port cc:8091/tcp -> 172.17.0.1:42461/tcp 2021-06-09T15:31:57.346-07:00 INFO DockerTestBase [main] binding exposed port cc:8095/tcp -> 172.17.0.1:44437/tcp 2021-06-09T15:31:57.399-07:00 INFO DockerTestBase [main] Starting docker container: 4b23412309e1c 2021-06-09T15:31:58.541-07:00 INFO DockerTestBase [main] Started docker container: 4b23412309e1c 2021-06-09T15:31:58.550-07:00 INFO DockerTestBase [main] Container information for 4b23412309e1c: host: cc.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:42461->192.168.176.3:8091, 172.17.0.1:44437->192.168.176.3:8095 2021-06-09T15:32:08.590-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {REBALANCE_REQUIRED} for up to 120s... 2021-06-09T15:32:17.234-07:00 INFO UpgradeTestBase [main] Recreating 'nc' node (image = null, forceNewIp = false)... 2021-06-09T15:32:17.235-07:00 INFO DockerTestBase [main] Copying files from docker container f7d9fde8e8fbe: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0003_nc_couchbase_enterprise-6.6.0 2021-06-09T15:32:17.782-07:00 INFO DockerTestBase [main] Stopping docker container: f7d9fde8e8fbe 2021-06-09T15:32:18.243-07:00 INFO DockerTestBase [main] Stopped docker container: f7d9fde8e8fbe 2021-06-09T15:32:18.243-07:00 INFO DockerTestBase [main] Removing docker container: f7d9fde8e8fbe 2021-06-09T15:32:18.254-07:00 INFO DockerTestBase [main] Removed docker container: f7d9fde8e8fbe 2021-06-09T15:32:18.255-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0006_nc_thisbuild, network = { name = cbas-docker-test-0001-eaa079a9-3fd9-4842-a280-0662d0e878d9, id = 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc }) 2021-06-09T15:32:18.256-07:00 INFO DockerTestBase [main] binding exposed port nc:8091/tcp -> 172.17.0.1:45503/tcp 2021-06-09T15:32:18.256-07:00 INFO DockerTestBase [main] binding exposed port nc:8095/tcp -> 172.17.0.1:33936/tcp 2021-06-09T15:32:18.316-07:00 INFO DockerTestBase [main] Starting docker container: 4af72e5a5c66c 2021-06-09T15:32:19.810-07:00 INFO DockerTestBase [main] Started docker container: 4af72e5a5c66c 2021-06-09T15:32:19.820-07:00 INFO DockerTestBase [main] Container information for 4af72e5a5c66c: host: nc.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:45503->192.168.176.4:8091, 172.17.0.1:33936->192.168.176.4:8095 2021-06-09T15:32:29.864-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {REBALANCE_REQUIRED} for up to 120s... 2021-06-09T15:32:29.907-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:32:31.151-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:32.155-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:32:33.161-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:32:34.165-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:32:35.169-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:36.172-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:37.176-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:38.181-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond 2021-06-09T15:32:39.184-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:40.188-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:41.192-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:42.195-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:43.199-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:44.202-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:32:45.205-07:00 WARN DockerTestBase [pool-2] unable to determine cluster compatibility: java.net.SocketException: Connection reset 2021-06-09T15:33:47.167-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:33:47.168-07:00 INFO DockerTestBase [main] Executing the command: docker exec kvnew.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.5:8091 -u couchbase -p couchbase 2021-06-09T15:33:54.627-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:33:54.627-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:33:54.649-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:33:54.659-07:00 INFO DockerTestBase [pool-3] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:33:54.659-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:33:54.659-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:33:54.736-07:00 INFO DockerTestBase [main] 0 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0002_cc_couchbase_enterprise-6.6.0/logs/analytics_info.log 2021-06-09T15:33:54.839-07:00 INFO DockerTestBase [main] 2 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0003_nc_couchbase_enterprise-6.6.0/logs/analytics_info.log 2021-06-09T15:33:54.840-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:42461] 2021-06-09T15:33:54.841-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:33:54.843-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:33:55.115-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:33:55.116-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:33:55.171-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:33:55.172-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:33:55.294-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:33:55.363-07:00 INFO UpgradeTestBase [main] Connecting link... 2021-06-09T15:33:55.439-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with values, nulls and missings... 2021-06-09T15:33:55.817-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:33:55.818-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:33:55.874-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:33:55.875-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:33:55.932-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:33:55.933-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:33:56.009-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:33:56.065-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with nulls 2021-06-09T15:33:56.080-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:33:56.081-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:33:56.163-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:33:56.164-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:33:56.227-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:33:56.227-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:33:56.293-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:33:56.347-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with missings 2021-06-09T15:33:56.369-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:33:56.370-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:33:56.421-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:33:56.421-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:33:56.474-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:33:56.475-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:33:56.566-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:33:56.626-07:00 INFO DockerTestBase [main] Copying files from docker container 4b23412309e1c: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0005_cc_thisbuild 2021-06-09T15:33:57.428-07:00 INFO DockerTestBase [main] Stopping docker container: 4b23412309e1c 2021-06-09T15:33:58.103-07:00 INFO DockerTestBase [main] Stopped docker container: 4b23412309e1c 2021-06-09T15:33:58.104-07:00 INFO DockerTestBase [main] Copying files from docker container 4af72e5a5c66c: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0006_nc_thisbuild 2021-06-09T15:33:58.916-07:00 INFO DockerTestBase [main] Stopping docker container: 4af72e5a5c66c 2021-06-09T15:33:59.357-07:00 INFO DockerTestBase [main] Stopped docker container: 4af72e5a5c66c 2021-06-09T15:33:59.421-07:00 INFO DockerTestBase [main] 0 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0005_cc_thisbuild/logs/analytics_info.log 2021-06-09T15:33:59.475-07:00 INFO DockerTestBase [main] 2 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0006_nc_thisbuild/logs/analytics_info.log 2021-06-09T15:33:59.475-07:00 INFO DockerTestBase [main] Starting docker container: 4af72e5a5c66c 2021-06-09T15:34:00.431-07:00 INFO DockerTestBase [main] Started docker container: 4af72e5a5c66c 2021-06-09T15:34:00.437-07:00 INFO DockerTestBase [main] Container information for 4af72e5a5c66c: host: nc.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.3, 172.17.0.1:45503->192.168.176.3:8091, 172.17.0.1:33936->192.168.176.3:8095 2021-06-09T15:34:00.444-07:00 INFO DockerTestBase [main] Copying files from docker container 0173b0d0bdd20: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0004_kvnew_thisbuild 2021-06-09T15:34:00.448-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Copying files from docker container 4af72e5a5c66c: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0006_nc_thisbuild 2021-06-09T15:34:00.448-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 62dc8b190510f: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOfflineAndDowngrade/0001_kv_couchbase_enterprise-6.6.0 2021-06-09T15:34:01.401-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopping docker container: 4af72e5a5c66c 2021-06-09T15:34:01.593-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 62dc8b190510f 2021-06-09T15:34:01.872-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Stopped docker container: 4af72e5a5c66c 2021-06-09T15:34:01.908-07:00 INFO DockerTestBase [main] Stopping docker container: 0173b0d0bdd20 2021-06-09T15:34:02.022-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 62dc8b190510f 2021-06-09T15:34:02.274-07:00 INFO DockerTestBase [main] Stopped docker container: 0173b0d0bdd20 2021-06-09T15:34:02.277-07:00 INFO DockerTestBase [main] Removing docker container: 4b23412309e1c 2021-06-09T15:34:02.279-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 0173b0d0bdd20 2021-06-09T15:34:02.279-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removing docker container: 4af72e5a5c66c 2021-06-09T15:34:02.279-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 62dc8b190510f 2021-06-09T15:34:02.294-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 62dc8b190510f 2021-06-09T15:34:02.296-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-7] Removed docker container: 4af72e5a5c66c 2021-06-09T15:34:02.298-07:00 INFO DockerTestBase [main] Removed docker container: 4b23412309e1c 2021-06-09T15:34:02.299-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 0173b0d0bdd20 2021-06-09T15:34:02.300-07:00 INFO DockerTestBase [main] Removing docker network: 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc 2021-06-09T15:34:02.369-07:00 INFO DockerTestBase [main] Removed docker network: 2be88f7c8c1f069e8e481ecc478dfbeac4e4fa018a759be10cf2df6526d18acc 2021-06-09T15:34:02.376-07:00 WARN TestMethodTracer [main] ### upgradeOfflineAndDowngrade SUCCEEDED 2021-06-09T15:34:02.377-07:00 WARN TestMethodTracer [main] ### upgradeOnline START 2021-06-09T15:34:02.378-07:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases. 2021-06-09T15:34:02.428-07:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0002-c8f9576c-c3f0-40de-92aa-8211d78c29c3) 2021-06-09T15:34:02.472-07:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0002-c8f9576c-c3f0-40de-92aa-8211d78c29c3, id = e93069f423be6489a4b04d844bb7bf0de10cdf3678c2c6ce82f460cf0f0c818a } 2021-06-09T15:34:02.473-07:00 INFO UpgradeTestBase [main] Creating n1 node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:34:02.474-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001n1 to /opt/couchbase/var 2021-06-09T15:34:02.474-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0007_n1_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0002-c8f9576c-c3f0-40de-92aa-8211d78c29c3, id = e93069f423be6489a4b04d844bb7bf0de10cdf3678c2c6ce82f460cf0f0c818a }) 2021-06-09T15:34:02.478-07:00 INFO DockerTestBase [main] binding exposed port n1:8091/tcp -> 172.17.0.1:39515/tcp 2021-06-09T15:34:02.478-07:00 INFO DockerTestBase [main] binding exposed port n1:8095/tcp -> 172.17.0.1:37024/tcp 2021-06-09T15:34:02.532-07:00 INFO DockerTestBase [main] Starting docker container: 3120efd4d6eaa 2021-06-09T15:34:04.532-07:00 INFO DockerTestBase [main] Started docker container: 3120efd4d6eaa 2021-06-09T15:34:04.572-07:00 INFO DockerTestBase [main] Container information for 3120efd4d6eaa: host: n1.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.2, 172.17.0.1:39515->192.168.176.2:8091, 172.17.0.1:37024->192.168.176.2:8095 2021-06-09T15:34:11.598-07:00 INFO DockerTestBase [main] Running 'node-init' on node n1 (192.168.176.2) 2021-06-09T15:34:11.598-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n1.couchbase.host 2021-06-09T15:34:12.351-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:34:12.377-07:00 INFO UpgradeTestBase [main] Creating n2 node ([couchbase:enterprise-6.6.0])... 2021-06-09T15:34:12.378-07:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002n2 to /opt/couchbase/var 2021-06-09T15:34:12.378-07:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0008_n2_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0002-c8f9576c-c3f0-40de-92aa-8211d78c29c3, id = e93069f423be6489a4b04d844bb7bf0de10cdf3678c2c6ce82f460cf0f0c818a }) 2021-06-09T15:34:12.379-07:00 INFO DockerTestBase [main] binding exposed port n2:8091/tcp -> 172.17.0.1:34130/tcp 2021-06-09T15:34:12.379-07:00 INFO DockerTestBase [main] binding exposed port n2:8095/tcp -> 172.17.0.1:41039/tcp 2021-06-09T15:34:12.427-07:00 INFO DockerTestBase [main] Starting docker container: 15ae252f12474 2021-06-09T15:34:14.051-07:00 INFO DockerTestBase [main] Started docker container: 15ae252f12474 2021-06-09T15:34:14.057-07:00 INFO DockerTestBase [main] Container information for 15ae252f12474: host: n2.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.3, 172.17.0.1:34130->192.168.176.3:8091, 172.17.0.1:41039->192.168.176.3:8095 2021-06-09T15:34:21.083-07:00 INFO DockerTestBase [main] Running 'node-init' on node n2 (192.168.176.3) 2021-06-09T15:34:21.084-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname n2.couchbase.host 2021-06-09T15:34:22.203-07:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized 2021-06-09T15:34:22.225-07:00 INFO DockerTestBase [main] Running 'cluster-init' on node n1 (192.168.176.2) with services: [DATA, ANALYTICS] 2021-06-09T15:34:22.225-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data,analytics 2021-06-09T15:34:22.602-07:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized 2021-06-09T15:34:22.638-07:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:34:35.601-07:00 INFO DockerTestBase [main] Running 'server-add' for node n2 (192.168.176.3) with services: [ANALYTICS] 2021-06-09T15:34:35.601-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://n2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:34:38.279-07:00 INFO DockerTestBase [main+] >> SUCCESS: Server added 2021-06-09T15:34:38.298-07:00 INFO UpgradeTestBase [main] Running 'rebalance' 2021-06-09T15:34:38.298-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase 2021-06-09T15:34:47.746-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:34:47.749-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:34:47.769-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:34:47.782-07:00 INFO DockerTestBase [pool-3] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:34:47.782-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:34:47.782-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:34:47.922-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:39515] 2021-06-09T15:34:47.923-07:00 INFO SecondaryCompositeIndexITD [main] creating bucket1 and bucket2... 2021-06-09T15:34:47.927-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket1) does not exist. 2021-06-09T15:34:47.934-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket1) created successfully. 2021-06-09T15:34:47.982-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket2) does not exist. 2021-06-09T15:34:48.008-07:00 INFO KvStoreHttpHelper [main] Bucket (bucket2) created successfully. 2021-06-09T15:34:48.048-07:00 INFO SecondaryCompositeIndexITD [main] creating datasets and indexes... 2021-06-09T15:34:49.495-07:00 INFO UpgradeTestBase [main] Connecting link... 2021-06-09T15:34:53.115-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with values, nulls and missings... 2021-06-09T15:34:53.323-07:00 INFO UpgradeTestBase [main] Upgrading [n1.couchbase.host] (online, rebalance) ... 2021-06-09T15:34:53.333-07:00 INFO UpgradeTestBase [pool-3] Creating n1-new node using 'this build' docker image... 2021-06-09T15:34:53.334-07:00 INFO UpgradeTestBase [pool-3] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003n1new to /opt/couchbase/var 2021-06-09T15:34:53.335-07:00 INFO DockerTestBase [pool-3] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0009_n1new_thisbuild, network = { name = cbas-docker-test-0002-c8f9576c-c3f0-40de-92aa-8211d78c29c3, id = e93069f423be6489a4b04d844bb7bf0de10cdf3678c2c6ce82f460cf0f0c818a }) 2021-06-09T15:34:53.336-07:00 INFO DockerTestBase [pool-3] binding exposed port n1new:8091/tcp -> 172.17.0.1:33776/tcp 2021-06-09T15:34:53.336-07:00 INFO DockerTestBase [pool-3] binding exposed port n1new:8095/tcp -> 172.17.0.1:41546/tcp 2021-06-09T15:34:53.405-07:00 INFO DockerTestBase [pool-3] Starting docker container: 283d1c25cb395 2021-06-09T15:34:55.197-07:00 INFO DockerTestBase [pool-3] Started docker container: 283d1c25cb395 2021-06-09T15:34:55.220-07:00 INFO DockerTestBase [pool-3] Container information for 283d1c25cb395: host: n1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.4, 172.17.0.1:33776->192.168.176.4:8091, 172.17.0.1:41546->192.168.176.4:8095 2021-06-09T15:35:06.806-07:00 INFO DockerTestBase [pool-3] Running 'node-init' on node n1new (192.168.176.4) 2021-06-09T15:35:06.807-07:00 INFO DockerTestBase [pool-3] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname n1new.couchbase.host 2021-06-09T15:35:07.877-07:00 INFO DockerTestBase [pool-3+] >> SUCCESS: Node initialized 2021-06-09T15:35:07.901-07:00 INFO DockerTestBase [pool-3] Running 'server-add' for node n1new (192.168.176.4) with services: [DATA, ANALYTICS] 2021-06-09T15:35:07.901-07:00 INFO DockerTestBase [pool-3] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://n1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data,analytics 2021-06-09T15:35:11.948-07:00 INFO DockerTestBase [pool-3+] >> SUCCESS: Server added 2021-06-09T15:35:11.973-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:35:11.973-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:35:11.973-07:00 INFO DockerTestBase [main] Executing the command: docker exec n2.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase --server-remove n1.couchbase.host 2021-06-09T15:38:23.998-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:38:23.998-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:38:24.029-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:38:24.042-07:00 INFO DockerTestBase [pool-5] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:38:24.043-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:38:24.044-07:00 INFO DockerTestBase [main] Copying files from docker container 3120efd4d6eaa: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0007_n1_couchbase_enterprise-6.6.0 2021-06-09T15:38:25.033-07:00 INFO DockerTestBase [main] Stopping docker container: 3120efd4d6eaa 2021-06-09T15:38:25.512-07:00 INFO DockerTestBase [main] Stopped docker container: 3120efd4d6eaa 2021-06-09T15:38:25.512-07:00 INFO DockerTestBase [main] Removing docker container: 3120efd4d6eaa 2021-06-09T15:38:25.522-07:00 INFO DockerTestBase [main] Removed docker container: 3120efd4d6eaa 2021-06-09T15:38:25.522-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:38:34.036-07:00 INFO UpgradeTestBase [main] Upgrading [n2.couchbase.host] (online, rebalance) ... 2021-06-09T15:38:34.036-07:00 INFO UpgradeTestBase [pool-5] Creating n2-new node using 'this build' docker image... 2021-06-09T15:38:34.036-07:00 INFO UpgradeTestBase [pool-5] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004n2new to /opt/couchbase/var 2021-06-09T15:38:34.037-07:00 INFO DockerTestBase [pool-5] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0010_n2new_thisbuild, network = { name = cbas-docker-test-0002-c8f9576c-c3f0-40de-92aa-8211d78c29c3, id = e93069f423be6489a4b04d844bb7bf0de10cdf3678c2c6ce82f460cf0f0c818a }) 2021-06-09T15:38:34.037-07:00 INFO DockerTestBase [pool-5] binding exposed port n2new:8091/tcp -> 172.17.0.1:37334/tcp 2021-06-09T15:38:34.038-07:00 INFO DockerTestBase [pool-5] binding exposed port n2new:8095/tcp -> 172.17.0.1:38252/tcp 2021-06-09T15:38:34.087-07:00 INFO DockerTestBase [pool-5] Starting docker container: 8ee7da0d96174 2021-06-09T15:38:35.426-07:00 INFO DockerTestBase [pool-5] Started docker container: 8ee7da0d96174 2021-06-09T15:38:35.439-07:00 INFO DockerTestBase [pool-5] Container information for 8ee7da0d96174: host: n2new.couchbase.host, image: analytics-jenkins-enterprise-7.0.1-1541690320, ip: 192.168.176.2, 172.17.0.1:37334->192.168.176.2:8091, 172.17.0.1:38252->192.168.176.2:8095 2021-06-09T15:38:45.645-07:00 INFO DockerTestBase [pool-5] Running 'node-init' on node n2new (192.168.176.2) 2021-06-09T15:38:45.645-07:00 INFO DockerTestBase [pool-5] Executing the command: docker exec n2new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname n2new.couchbase.host 2021-06-09T15:38:46.408-07:00 INFO DockerTestBase [pool-5+] >> SUCCESS: Node initialized 2021-06-09T15:38:46.433-07:00 INFO DockerTestBase [pool-5] Running 'server-add' for node n2new (192.168.176.2) with services: [ANALYTICS] 2021-06-09T15:38:46.433-07:00 INFO DockerTestBase [pool-5] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://n2new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics 2021-06-09T15:38:50.510-07:00 INFO DockerTestBase [pool-5+] >> SUCCESS: Server added 2021-06-09T15:38:50.538-07:00 INFO UpgradeTestBase [main] Swap rebalancing ... 2021-06-09T15:38:50.538-07:00 INFO DockerTestBase [main] Running 'rebalance' 2021-06-09T15:38:50.538-07:00 INFO DockerTestBase [main] Executing the command: docker exec n1new.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove n2.couchbase.host 2021-06-09T15:39:03.142-07:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os 2021-06-09T15:39:03.142-07:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete 2021-06-09T15:39:03.170-07:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds... 2021-06-09T15:39:03.188-07:00 INFO DockerTestBase [pool-5] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE 2021-06-09T15:39:03.189-07:00 INFO UpgradeTestBase [main] ... rebalance succeeded! 2021-06-09T15:39:03.189-07:00 INFO DockerTestBase [main] Copying files from docker container 15ae252f12474: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0008_n2_couchbase_enterprise-6.6.0 2021-06-09T15:39:03.760-07:00 INFO DockerTestBase [main] Stopping docker container: 15ae252f12474 2021-06-09T15:39:04.144-07:00 INFO DockerTestBase [main] Stopped docker container: 15ae252f12474 2021-06-09T15:39:04.144-07:00 INFO DockerTestBase [main] Removing docker container: 15ae252f12474 2021-06-09T15:39:04.151-07:00 INFO DockerTestBase [main] Removed docker container: 15ae252f12474 2021-06-09T15:39:04.152-07:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s... 2021-06-09T15:39:17.975-07:00 INFO DockerTestBase [main] 1 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0007_n1_couchbase_enterprise-6.6.0/logs/analytics_info.log 2021-06-09T15:39:17.988-07:00 INFO DockerTestBase [main] 0 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0008_n2_couchbase_enterprise-6.6.0/logs/analytics_info.log 2021-06-09T15:39:17.995-07:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds... 2021-06-09T15:39:18.257-07:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:33776] 2021-06-09T15:39:18.258-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:39:18.258-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:39:18.855-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:39:18.856-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:39:18.933-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:39:18.934-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:39:19.246-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:39:19.326-07:00 INFO UpgradeTestBase [main] Connecting link... 2021-06-09T15:39:21.184-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with values, nulls and missings... 2021-06-09T15:39:21.587-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:39:21.588-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:39:21.839-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:39:21.840-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:39:22.008-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:39:22.008-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:39:22.159-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:39:22.447-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with nulls 2021-06-09T15:39:22.474-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:39:22.474-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:39:22.666-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:39:22.667-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:39:22.841-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:39:22.844-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:39:22.930-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:39:22.984-07:00 INFO SecondaryCompositeIndexITD [main] upserting documents with missings 2021-06-09T15:39:22.997-07:00 INFO SecondaryCompositeIndexITD [main] checking ds1 2021-06-09T15:39:22.998-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE 2021-06-09T15:39:23.094-07:00 INFO SecondaryCompositeIndexITD [main] checking ds2 2021-06-09T15:39:23.095-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE 2021-06-09T15:39:23.175-07:00 INFO SecondaryCompositeIndexITD [main] checking idx1 2021-06-09T15:39:23.175-07:00 INFO TestExecutor [main] using Accept-Charset: UTF-32 2021-06-09T15:39:23.242-07:00 INFO SecondaryCompositeIndexITD [main] checking idx2 2021-06-09T15:39:23.302-07:00 INFO DockerTestBase [main] Copying files from docker container 283d1c25cb395: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0009_n1new_thisbuild 2021-06-09T15:39:24.794-07:00 INFO DockerTestBase [main] Stopping docker container: 283d1c25cb395 2021-06-09T15:39:25.397-07:00 INFO DockerTestBase [main] Stopped docker container: 283d1c25cb395 2021-06-09T15:39:25.397-07:00 INFO DockerTestBase [main] Copying files from docker container 8ee7da0d96174: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0010_n2new_thisbuild 2021-06-09T15:39:25.792-07:00 INFO DockerTestBase [main] Stopping docker container: 8ee7da0d96174 2021-06-09T15:39:26.323-07:00 INFO DockerTestBase [main] Stopped docker container: 8ee7da0d96174 2021-06-09T15:39:26.332-07:00 INFO DockerTestBase [main] 0 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0009_n1new_thisbuild/logs/analytics_info.log 2021-06-09T15:39:26.336-07:00 INFO DockerTestBase [main] 0 occurrence(s) of pattern .*at org\.apache\.asterix\.dataflow\.data\.nontagged\.comparators\.AGenericAscBinaryComparator\.compare.* in file /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD/upgradeOnline/0010_n2new_thisbuild/logs/analytics_info.log 2021-06-09T15:39:26.336-07:00 INFO DockerTestBase [main] Removing docker container: 8ee7da0d96174 2021-06-09T15:39:26.336-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removing docker container: 283d1c25cb395 2021-06-09T15:39:26.346-07:00 INFO DockerTestBase [main] Removed docker container: 8ee7da0d96174 2021-06-09T15:39:26.346-07:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Removed docker container: 283d1c25cb395 2021-06-09T15:39:26.347-07:00 INFO DockerTestBase [main] Removing docker network: e93069f423be6489a4b04d844bb7bf0de10cdf3678c2c6ce82f460cf0f0c818a 2021-06-09T15:39:26.399-07:00 INFO DockerTestBase [main] Removed docker network: e93069f423be6489a4b04d844bb7bf0de10cdf3678c2c6ce82f460cf0f0c818a 2021-06-09T15:39:26.401-07:00 WARN TestMethodTracer [main] ### upgradeOnline SUCCEEDED [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 745.348 s - in com.couchbase.analytics.test.docker.upgrade.misc.SecondaryCompositeIndexITD [INFO] [INFO] Results: [INFO] [ERROR] Errors: [ERROR] QuorumFailoverITD>ClusterTestBase.lambda$invokeAll$38:1253->UpgradeTestBase.createNode:160->UpgradeTestBase.createNode:164->UpgradeTestBase.createNode:170->DockerTestBase.createAndStartContainer:415->DockerTestBase.createAndStartContainer:421->DockerTestBase.createAndStartContainer:428->DockerTestBase.createAndStartContainer:453->DockerTestBase.createAndStartContainer:577->ClusterTestBase.waitForStatus:858->DockerTestBase.getBasicAuthHttpCtx:239 ยป ConcurrentModification [INFO] [ERROR] Tests run: 101, Failures: 0, Errors: 1, Skipped: 1 [INFO] [JENKINS] Recording test results [INFO] [INFO] --- maven-failsafe-plugin:2.21.0:verify (default) @ cbas-docker-test --- [ERROR] There are test failures. Please refer to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/failsafe-reports for the individual test results. Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream. [JENKINS] Recording test results [JENKINS] Archiving disabled