2021-12-05T12:56:44.811-08:00 INFO ClusterTestBase [main] configured sslWorkDir to /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/sslWork
|
2021-12-05T12:56:44.840-08:00 INFO DockerTestBase [main] resolved dockerHostAddr using 'host.docker.internal': 172.17.0.1
|
2021-12-05T12:56:44.841-08:00 INFO DockerTestBase [main] Docker image id=build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.3-1669910388 dockerHostAddr=172.17.0.1 dockerContainerId=55fb6de40a62c1d053270b31cf4fb2414ca149423fc956ea85ee2814101f59ed
|
2021-12-05T12:56:44.955-08:00 WARN TestMethodTracer [main] ### failoverUpgradeAnalytics START
|
2021-12-05T12:56:44.987-08:00 WARN DockerClientBuilder [main] 'dockerHttpClient' should be set.Falling back to Jersey, will be an error in future releases.
|
2021-12-05T12:56:45.851-08:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864)
|
2021-12-05T12:56:45.928-08:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864, id = 45185a0088d767659955e76735e0e0f9417dd42a325c1d618dfb3bd38b2bfa82 }
|
2021-12-05T12:56:45.932-08:00 INFO UpgradeTestBase [pool-0] Creating kv1 node ([couchbase:enterprise-6.6.0])...
|
2021-12-05T12:56:45.932-08:00 INFO UpgradeTestBase [pool-1] Creating cbas1 node ([couchbase:enterprise-6.6.0])...
|
2021-12-05T12:56:45.937-08:00 INFO UpgradeTestBase [pool-0] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv1 to /opt/couchbase/var
|
2021-12-05T12:56:45.937-08:00 INFO UpgradeTestBase [pool-1] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002cbas1 to /opt/couchbase/var
|
2021-12-05T12:56:45.933-08:00 INFO UpgradeTestBase [pool-2] Creating cbas2 node ([couchbase:enterprise-6.6.0])...
|
2021-12-05T12:56:45.943-08:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003cbas2 to /opt/couchbase/var
|
2021-12-05T12:56:45.943-08:00 INFO DockerTestBase [pool-1] Executing the command: docker pull couchbase:enterprise-6.6.0
|
2021-12-05T12:56:46.976-08:00 INFO DockerTestBase [pool-1+] >> enterprise-6.6.0: Pulling from library/couchbase
|
2021-12-05T12:56:46.981-08:00 INFO DockerTestBase [pool-1+] >> Digest: sha256:6b1d9cf03e16f6a4095c6ad367d0fa4e745e56825f794e3f0f6af9365f5eb01a
|
2021-12-05T12:56:46.981-08:00 INFO DockerTestBase [pool-1+] >> Status: Image is up to date for couchbase:enterprise-6.6.0
|
2021-12-05T12:56:46.982-08:00 INFO DockerTestBase [pool-1+] >> docker.io/library/couchbase:enterprise-6.6.0
|
2021-12-05T12:56:46.985-08:00 INFO DockerTestBase [pool-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0001_cbas1_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864, id = 45185a0088d767659955e76735e0e0f9417dd42a325c1d618dfb3bd38b2bfa82 })
|
2021-12-05T12:56:46.986-08:00 INFO DockerTestBase [pool-0] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0002_kv1_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864, id = 45185a0088d767659955e76735e0e0f9417dd42a325c1d618dfb3bd38b2bfa82 })
|
2021-12-05T12:56:46.989-08:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0003_cbas2_couchbase_enterprise-6.6.0, network = { name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864, id = 45185a0088d767659955e76735e0e0f9417dd42a325c1d618dfb3bd38b2bfa82 })
|
2021-12-05T12:56:47.015-08:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8091/tcp -> 172.17.0.1:40621/tcp
|
2021-12-05T12:56:47.015-08:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8091/tcp -> 172.17.0.1:43987/tcp
|
2021-12-05T12:56:47.015-08:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8091/tcp -> 172.17.0.1:41327/tcp
|
2021-12-05T12:56:47.015-08:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:8095/tcp -> 172.17.0.1:42395/tcp
|
2021-12-05T12:56:47.016-08:00 INFO DockerTestBase [pool-1] binding exposed port cbas1:11210/tcp -> 172.17.0.1:46865/tcp
|
2021-12-05T12:56:47.016-08:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:8095/tcp -> 172.17.0.1:36941/tcp
|
2021-12-05T12:56:47.017-08:00 INFO DockerTestBase [pool-2] binding exposed port cbas2:11210/tcp -> 172.17.0.1:41079/tcp
|
2021-12-05T12:56:47.018-08:00 INFO DockerTestBase [pool-0] binding exposed port kv1:8095/tcp -> 172.17.0.1:36469/tcp
|
2021-12-05T12:56:47.019-08:00 INFO DockerTestBase [pool-0] binding exposed port kv1:11210/tcp -> 172.17.0.1:36741/tcp
|
2021-12-05T12:56:47.181-08:00 INFO DockerTestBase [pool-1] Starting docker container: 7053681ffa5c5
|
2021-12-05T12:56:47.202-08:00 INFO DockerTestBase [pool-0] Starting docker container: d40ca4e98924f
|
2021-12-05T12:56:47.212-08:00 INFO DockerTestBase [pool-2] Starting docker container: 142f9d7e28cce
|
2021-12-05T12:56:48.672-08:00 INFO DockerTestBase [pool-1] Started docker container: 7053681ffa5c5
|
2021-12-05T12:56:48.787-08:00 INFO DockerTestBase [pool-1] Container information for 7053681ffa5c5: host: cbas1.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.2, 172.17.0.1:42395->192.168.176.2:8095, 172.17.0.1:46865->192.168.176.2:11210, 172.17.0.1:40621->192.168.176.2:8091
|
2021-12-05T12:56:49.297-08:00 INFO DockerTestBase [pool-2] Started docker container: 142f9d7e28cce
|
2021-12-05T12:56:49.304-08:00 INFO DockerTestBase [pool-2] Container information for 142f9d7e28cce: host: cbas2.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.4, 172.17.0.1:36941->192.168.176.4:8095, 172.17.0.1:41079->192.168.176.4:11210, 172.17.0.1:41327->192.168.176.4:8091
|
2021-12-05T12:56:50.010-08:00 INFO DockerTestBase [pool-0] Started docker container: d40ca4e98924f
|
2021-12-05T12:56:50.020-08:00 INFO DockerTestBase [pool-0] Container information for d40ca4e98924f: host: kv1.couchbase.host, image: enterprise-6.6.0, ip: 192.168.176.3, 172.17.0.1:36469->192.168.176.3:8095, 172.17.0.1:36741->192.168.176.3:11210, 172.17.0.1:43987->192.168.176.3:8091
|
2021-12-05T12:56:58.897-08:00 INFO DockerTestBase [pool-1] Running 'node-init' on node cbas1 (192.168.176.2)
|
2021-12-05T12:56:58.899-08:00 INFO DockerTestBase [pool-1] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname cbas1.couchbase.host
|
2021-12-05T12:56:59.345-08:00 INFO DockerTestBase [pool-2] Running 'node-init' on node cbas2 (192.168.176.4)
|
2021-12-05T12:56:59.346-08:00 INFO DockerTestBase [pool-2] Executing the command: docker exec cbas2.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname cbas2.couchbase.host
|
2021-12-05T12:56:59.923-08:00 INFO DockerTestBase [pool-1+] >> SUCCESS: Node initialized
|
2021-12-05T12:57:00.066-08:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized
|
2021-12-05T12:57:00.123-08:00 INFO DockerTestBase [pool-0] Running 'node-init' on node kv1 (192.168.176.3)
|
2021-12-05T12:57:00.124-08:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname kv1.couchbase.host
|
2021-12-05T12:57:00.665-08:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Node initialized
|
2021-12-05T12:57:00.689-08:00 INFO DockerTestBase [pool-0] Running 'cluster-init' on node kv1 (192.168.176.3) with services: [DATA]
|
2021-12-05T12:57:00.689-08:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data
|
2021-12-05T12:57:01.006-08:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Cluster initialized
|
2021-12-05T12:57:01.034-08:00 INFO DockerTestBase [pool-0] Running 'server-add' for node cbas1 (192.168.176.2) with services: [ANALYTICS]
|
2021-12-05T12:57:01.035-08:00 INFO DockerTestBase [pool-0] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://cbas1.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics
|
2021-12-05T12:57:01.034-08:00 INFO DockerTestBase [pool-5] Running 'server-add' for node cbas2 (192.168.176.4) with services: [ANALYTICS]
|
2021-12-05T12:57:01.036-08:00 INFO DockerTestBase [pool-5] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://cbas2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics
|
2021-12-05T12:57:04.004-08:00 INFO DockerTestBase [pool-0+] >> SUCCESS: Server added
|
2021-12-05T12:57:06.514-08:00 INFO DockerTestBase [pool-5+] >> SUCCESS: Server added
|
2021-12-05T12:57:06.545-08:00 INFO UpgradeTestBase [main] Running 'rebalance'
|
2021-12-05T12:57:06.546-08:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase
|
2021-12-05T12:57:07.863-08:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
|
2021-12-05T12:57:07.864-08:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete
|
2021-12-05T12:57:07.897-08:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds...
|
2021-12-05T12:57:07.918-08:00 INFO DockerTestBase [pool-5] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE
|
2021-12-05T12:57:07.918-08:00 INFO UpgradeTestBase [main] ... rebalance succeeded!
|
2021-12-05T12:57:07.925-08:00 INFO DockerTestBase [pool-0] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s...
|
2021-12-05T12:57:07.957-08:00 INFO KvStoreHttpHelper [pool-5] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:40621]
|
2021-12-05T12:57:07.963-08:00 INFO KvStoreHttpHelper [pool-5] installing sample(s): [beer-sample] on 172.17.0.1:40621
|
2021-12-05T12:57:16.190-08:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:40621]
|
2021-12-05T12:57:18.260-08:00 INFO DockerTestBase [main] Waiting for all beers to be ingested...
|
2021-12-05T12:57:18.262-08:00 INFO TestExecutor [main] using Accept-Charset: UTF-16LE
|
2021-12-05T12:57:18.762-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 0 }
|
2021-12-05T12:57:18.765-08:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 0 }
|
2021-12-05T12:57:19.765-08:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE
|
2021-12-05T12:57:20.036-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2936 }
|
2021-12-05T12:57:20.038-08:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 2936 }
|
2021-12-05T12:57:21.038-08:00 INFO TestExecutor [main] using Accept-Charset: UTF-32
|
2021-12-05T12:57:21.152-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 5891 }
|
2021-12-05T12:57:21.153-08:00 INFO DockerTestBase [main] Success
|
2021-12-05T12:57:21.153-08:00 INFO UpgradeFailoverTestBase [main] Failing over (hard) cbas nodes ...
|
2021-12-05T12:57:21.165-08:00 INFO DockerTestBase [main] Running 'failover --hard' on [cbas1 (192.168.176.2), cbas2 (192.168.176.4)] nodes
|
2021-12-05T12:57:21.166-08:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli failover -c 192.168.176.3:8091 -u couchbase -p couchbase --server-failover cbas1.couchbase.host,cbas2.couchbase.host --hard
|
2021-12-05T12:57:21.482-08:00 INFO DockerTestBase [main+] >> SUCCESS: Server failed over
|
2021-12-05T12:57:26.594-08:00 INFO UpgradeTestBase [main] Recreating 'cbas2' node (image = null, forceNewIp = false)...
|
2021-12-05T12:57:26.594-08:00 INFO DockerTestBase [main] Copying files from docker container 142f9d7e28cce: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0003_cbas2_couchbase_enterprise-6.6.0
|
2021-12-05T12:57:26.594-08:00 INFO UpgradeTestBase [ForkJoinPool.commonPool-worker-3] Recreating 'cbas1' node (image = null, forceNewIp = false)...
|
2021-12-05T12:57:26.595-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Copying files from docker container 7053681ffa5c5: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0001_cbas1_couchbase_enterprise-6.6.0
|
2021-12-05T12:57:26.858-08:00 INFO DockerTestBase [main] Stopping docker container: 142f9d7e28cce
|
2021-12-05T12:57:26.873-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopping docker container: 7053681ffa5c5
|
2021-12-05T12:57:27.484-08:00 INFO DockerTestBase [main] Stopped docker container: 142f9d7e28cce
|
2021-12-05T12:57:27.484-08:00 INFO DockerTestBase [main] Removing docker container: 142f9d7e28cce
|
2021-12-05T12:57:27.494-08:00 INFO DockerTestBase [main] Removed docker container: 142f9d7e28cce
|
2021-12-05T12:57:27.496-08:00 INFO DockerTestBase [main] Executing the command: docker pull build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.3-1669910388
|
2021-12-05T12:57:27.540-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Stopped docker container: 7053681ffa5c5
|
2021-12-05T12:57:27.540-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removing docker container: 7053681ffa5c5
|
2021-12-05T12:57:27.553-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Removed docker container: 7053681ffa5c5
|
2021-12-05T12:57:27.577-08:00 INFO DockerTestBase [main+] >> analytics-jenkins-enterprise-7.0.3-1669910388: Pulling from couchbase/server-toy
|
2021-12-05T12:57:27.578-08:00 INFO DockerTestBase [main+] >> Digest: sha256:d3b83eda10c26d69846d11731affd0a5f433e4ebe1868797a251d6329c742f43
|
2021-12-05T12:57:27.578-08:00 INFO DockerTestBase [main+] >> Status: Image is up to date for build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.3-1669910388
|
2021-12-05T12:57:27.580-08:00 INFO DockerTestBase [main+] >> build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.3-1669910388
|
2021-12-05T12:57:27.582-08:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0004_cbas2_thisbuild, network = { name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864, id = 45185a0088d767659955e76735e0e0f9417dd42a325c1d618dfb3bd38b2bfa82 })
|
2021-12-05T12:57:27.582-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0005_cbas1_thisbuild, network = { name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864, id = 45185a0088d767659955e76735e0e0f9417dd42a325c1d618dfb3bd38b2bfa82 })
|
2021-12-05T12:57:27.583-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8091/tcp -> 172.17.0.1:35815/tcp
|
2021-12-05T12:57:27.583-08:00 INFO DockerTestBase [main] binding exposed port cbas2:8091/tcp -> 172.17.0.1:38865/tcp
|
2021-12-05T12:57:27.584-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:8095/tcp -> 172.17.0.1:34747/tcp
|
2021-12-05T12:57:27.584-08:00 INFO DockerTestBase [main] binding exposed port cbas2:8095/tcp -> 172.17.0.1:46025/tcp
|
2021-12-05T12:57:27.584-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] binding exposed port cbas1:11210/tcp -> 172.17.0.1:45105/tcp
|
2021-12-05T12:57:27.584-08:00 INFO DockerTestBase [main] binding exposed port cbas2:11210/tcp -> 172.17.0.1:41911/tcp
|
2021-12-05T12:57:27.639-08:00 INFO DockerTestBase [main] Starting docker container: 101f96678fd90
|
2021-12-05T12:57:27.642-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Starting docker container: bc873f7150ae1
|
2021-12-05T12:57:29.008-08:00 INFO DockerTestBase [main] Started docker container: 101f96678fd90
|
2021-12-05T12:57:29.017-08:00 INFO DockerTestBase [main] Container information for 101f96678fd90: host: cbas2.couchbase.host, image: analytics-jenkins-enterprise-7.0.3-1669910388, ip: 192.168.176.2, 172.17.0.1:46025->192.168.176.2:8095, 172.17.0.1:41911->192.168.176.2:11210, 172.17.0.1:38865->192.168.176.2:8091
|
2021-12-05T12:57:30.309-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Started docker container: bc873f7150ae1
|
2021-12-05T12:57:30.317-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Container information for bc873f7150ae1: host: cbas1.couchbase.host, image: analytics-jenkins-enterprise-7.0.3-1669910388, ip: 192.168.176.4, 172.17.0.1:34747->192.168.176.4:8095, 172.17.0.1:45105->192.168.176.4:11210, 172.17.0.1:35815->192.168.176.4:8091
|
2021-12-05T12:57:43.358-08:00 INFO DockerTestBase [main] Running full 'recovery' on nodes [cbas1 (192.168.176.2), cbas2 (192.168.176.4)]
|
2021-12-05T12:57:43.359-08:00 INFO DockerTestBase [main] Executing the command: docker exec kv1.couchbase.host /opt/couchbase/bin/couchbase-cli recovery -c 192.168.176.3:8091 -u couchbase -p couchbase --server-recovery cbas1.couchbase.host,cbas2.couchbase.host --recovery-type full
|
2021-12-05T12:57:43.660-08:00 INFO DockerTestBase [main+] >> SUCCESS: Servers recovered
|
2021-12-05T12:57:43.698-08:00 INFO UpgradeTestBase [main] Upgrading [kv1.couchbase.host] (online, rebalance) to version [build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.3-1669910388] ...
|
2021-12-05T12:57:43.700-08:00 INFO UpgradeTestBase [pool-2] Creating kv1-new node ([build-docker.couchbase.com/couchbase/server-toy:analytics-jenkins-enterprise-7.0.3-1669910388])...
|
2021-12-05T12:57:43.702-08:00 INFO UpgradeTestBase [pool-2] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004kv1new to /opt/couchbase/var
|
2021-12-05T12:57:43.703-08:00 INFO DockerTestBase [pool-2] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0006_kv1new_thisbuild, network = { name = cbas-docker-test-0001-75f11669-524a-45d1-9a84-d97ca4c8b864, id = 45185a0088d767659955e76735e0e0f9417dd42a325c1d618dfb3bd38b2bfa82 })
|
2021-12-05T12:57:43.703-08:00 INFO DockerTestBase [pool-2] binding exposed port kv1new:8091/tcp -> 172.17.0.1:34653/tcp
|
2021-12-05T12:57:43.705-08:00 INFO DockerTestBase [pool-2] binding exposed port kv1new:8095/tcp -> 172.17.0.1:37501/tcp
|
2021-12-05T12:57:43.705-08:00 INFO DockerTestBase [pool-2] binding exposed port kv1new:11210/tcp -> 172.17.0.1:43639/tcp
|
2021-12-05T12:57:43.791-08:00 INFO DockerTestBase [pool-2] Starting docker container: 37b71ce665472
|
2021-12-05T12:57:45.031-08:00 INFO DockerTestBase [pool-2] Started docker container: 37b71ce665472
|
2021-12-05T12:57:45.036-08:00 INFO DockerTestBase [pool-2] Container information for 37b71ce665472: host: kv1new.couchbase.host, image: analytics-jenkins-enterprise-7.0.3-1669910388, ip: 192.168.176.5, 172.17.0.1:37501->192.168.176.5:8095, 172.17.0.1:43639->192.168.176.5:11210, 172.17.0.1:34653->192.168.176.5:8091
|
2021-12-05T12:57:58.070-08:00 INFO DockerTestBase [pool-2] Running 'node-init' on node kv1new (192.168.176.5)
|
2021-12-05T12:57:58.071-08:00 INFO DockerTestBase [pool-2] Executing the command: docker exec kv1new.couchbase.host /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname kv1new.couchbase.host
|
2021-12-05T12:57:58.844-08:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Node initialized
|
2021-12-05T12:57:58.869-08:00 INFO DockerTestBase [pool-2] Running 'server-add' for node kv1new (192.168.176.5) with services: [DATA]
|
2021-12-05T12:57:58.870-08:00 INFO DockerTestBase [pool-2] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.4:8091 -u couchbase -p couchbase --server-add http://kv1new.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data
|
2021-12-05T12:58:03.457-08:00 INFO DockerTestBase [pool-2+] >> SUCCESS: Server added
|
2021-12-05T12:58:03.484-08:00 INFO UpgradeTestBase [main] Swap rebalancing ...
|
2021-12-05T12:58:03.484-08:00 INFO DockerTestBase [main] Running 'rebalance'
|
2021-12-05T12:58:03.485-08:00 INFO DockerTestBase [main] Executing the command: docker exec cbas1.couchbase.host /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.4:8091 -u couchbase -p couchbase --server-remove kv1.couchbase.host
|
2021-12-05T12:59:25.845-08:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
|
2021-12-05T12:59:25.846-08:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete
|
2021-12-05T12:59:25.876-08:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds...
|
2021-12-05T12:59:25.907-08:00 INFO DockerTestBase [pool-6] rebalance tasks complete; waiting for all analytics nodes & cluster state ACTIVE
|
2021-12-05T12:59:25.908-08:00 INFO UpgradeTestBase [main] ... rebalance succeeded!
|
2021-12-05T12:59:25.909-08:00 INFO DockerTestBase [main] Copying files from docker container d40ca4e98924f: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-upgrade-madhatter/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.madhatter.UpgradeFrom660FailoverITD/failoverUpgradeAnalytics/0002_kv1_couchbase_enterprise-6.6.0
|
2021-12-05T12:59:26.460-08:00 INFO DockerTestBase [main] Stopping docker container: d40ca4e98924f
|
2021-12-05T12:59:27.067-08:00 INFO DockerTestBase [main] Stopped docker container: d40ca4e98924f
|
2021-12-05T12:59:27.067-08:00 INFO DockerTestBase [main] Removing docker container: d40ca4e98924f
|
2021-12-05T12:59:27.075-08:00 INFO DockerTestBase [main] Removed docker container: d40ca4e98924f
|
2021-12-05T12:59:27.082-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds...
|
2021-12-05T12:59:27.083-08:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds...
|
2021-12-05T13:00:40.863-08:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s...
|
2021-12-05T13:00:40.864-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 120s...
|
2021-12-05T13:00:42.050-08:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:38865]
|
2021-12-05T13:00:42.050-08:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-3] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:35815]
|
2021-12-05T13:00:42.050-08:00 INFO DockerTestBase [main] Waiting for all beers to be ingested...
|
2021-12-05T13:00:42.050-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Waiting for all beers to be ingested...
|
2021-12-05T13:00:42.051-08:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16LE
|
2021-12-05T13:00:42.719-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:42.719-08:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:42.731-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:42.732-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 1. ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:43.720-08:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE
|
2021-12-05T13:00:43.732-08:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-32
|
2021-12-05T13:00:44.138-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:44.138-08:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:44.166-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:44.166-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 2. ResultSet found : { "$1": 1598 }
|
2021-12-05T13:00:45.166-08:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16LE
|
2021-12-05T13:00:45.579-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 2967 }
|
2021-12-05T13:00:45.579-08:00 INFO DockerTestBase [main] Failed attempt number: 3. ResultSet found : { "$1": 2967 }
|
2021-12-05T13:00:45.589-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 2967 }
|
2021-12-05T13:00:45.589-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 3. ResultSet found : { "$1": 2967 }
|
2021-12-05T13:00:46.579-08:00 INFO TestExecutor [main] using Accept-Charset: UTF-16BE
|
2021-12-05T13:00:46.589-08:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-32
|
2021-12-05T13:00:46.684-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 7489 }
|
2021-12-05T13:00:46.684-08:00 INFO DockerTestBase [main] Failed attempt number: 4. ResultSet found : { "$1": 7489 }
|
2021-12-05T13:00:46.690-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 7489 }
|
2021-12-05T13:00:46.690-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 4. ResultSet found : { "$1": 7489 }
|
2021-12-05T13:00:47.691-08:00 INFO TestExecutor [ForkJoinPool.commonPool-worker-3] using Accept-Charset: UTF-16LE
|
2021-12-05T13:00:47.807-08:00 INFO DockerTestBase [main] ResultSet found : { "$1": 7489 }
|
2021-12-05T13:00:47.807-08:00 INFO DockerTestBase [main] Failed attempt number: 5. ResultSet found : { "$1": 7489 }
|
2021-12-05T13:00:47.817-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] ResultSet found : { "$1": 7489 }
|
2021-12-05T13:00:47.817-08:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-3] Failed attempt number: 5. ResultSet found : { "$1": 7489 }
|
Task
Critical