2020-06-22T15:41:33.878+00:00 WARN TestMethodTracer [main] ### failoverUpgradeAll START
2020-06-22T15:41:33.916+00:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001)
2020-06-22T15:41:34.135+00:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e }
2020-06-22T15:41:34.135+00:00 INFO UpgradeTestBase [main] Creating kv1 node ...
2020-06-22T15:41:34.136+00:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv1 to /opt/couchbase/var
2020-06-22T15:41:34.136+00:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0009_kv1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:41:35.297+00:00 INFO DockerTestBase [main] Container information for fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230: host: kv1.couchbase.host, ip: 192.168.176.2, 172.19.0.1:41273->192.168.176.2:8095, 172.19.0.1:41277->192.168.176.2:8091
2020-06-22T15:41:44.330+00:00 INFO DockerTestBase [main] Running 'node-init' on node kv1 (192.168.176.2)
2020-06-22T15:41:44.331+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv1.couchbase.host
2020-06-22T15:41:44.900+00:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized
2020-06-22T15:41:44.905+00:00 INFO DockerTestBase [main] Running 'cluster-init' on node kv1 (192.168.176.2) with services: [DATA]
2020-06-22T15:41:44.905+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data
2020-06-22T15:41:45.140+00:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized
2020-06-22T15:41:45.146+00:00 INFO UpgradeTestBase [main] Creating kv2 node ...
2020-06-22T15:41:45.146+00:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002kv2 to /opt/couchbase/var
2020-06-22T15:41:45.146+00:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0010_kv2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:41:46.416+00:00 INFO DockerTestBase [main] Container information for e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059: host: kv2.couchbase.host, ip: 192.168.176.3, 172.19.0.1:41288->192.168.176.3:8095, 172.19.0.1:41292->192.168.176.3:8091
2020-06-22T15:41:55.460+00:00 INFO DockerTestBase [main] Running 'node-init' on node kv2 (192.168.176.3)
2020-06-22T15:41:55.460+00:00 INFO DockerTestBase [main] Executing the command: docker exec e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059 /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname kv2.couchbase.host
2020-06-22T15:41:56.050+00:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized
2020-06-22T15:41:56.056+00:00 INFO UpgradeTestBase [main] Creating cbas1 node ...
2020-06-22T15:41:56.056+00:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003cbas1 to /opt/couchbase/var
2020-06-22T15:41:56.057+00:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0011_cbas1_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:41:57.973+00:00 INFO DockerTestBase [main] Container information for 26b2aee89c095b437f49e53f4142002e7c7632f9765b40726e24c54b183b0c5e: host: cbas1.couchbase.host, ip: 192.168.176.4, 172.19.0.1:41303->192.168.176.4:8095, 172.19.0.1:41307->192.168.176.4:8091
2020-06-22T15:42:07.012+00:00 INFO DockerTestBase [main] Running 'node-init' on node cbas1 (192.168.176.4)
2020-06-22T15:42:07.012+00:00 INFO DockerTestBase [main] Executing the command: docker exec 26b2aee89c095b437f49e53f4142002e7c7632f9765b40726e24c54b183b0c5e /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname cbas1.couchbase.host
2020-06-22T15:42:07.610+00:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized
2020-06-22T15:42:07.618+00:00 INFO UpgradeTestBase [main] Creating cbas2 node ...
2020-06-22T15:42:07.618+00:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0004cbas2 to /opt/couchbase/var
2020-06-22T15:42:07.619+00:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0012_cbas2_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:42:09.487+00:00 INFO DockerTestBase [main] Container information for 91d28638f657427c81a250d265cf2d49f4b80ee85afd5404643b9fb2e0bc3526: host: cbas2.couchbase.host, ip: 192.168.176.5, 172.19.0.1:41318->192.168.176.5:8095, 172.19.0.1:41322->192.168.176.5:8091
2020-06-22T15:42:18.526+00:00 INFO DockerTestBase [main] Running 'node-init' on node cbas2 (192.168.176.5)
2020-06-22T15:42:18.526+00:00 INFO DockerTestBase [main] Executing the command: docker exec 91d28638f657427c81a250d265cf2d49f4b80ee85afd5404643b9fb2e0bc3526 /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.5:8091 -u couchbase -p couchbase --node-init-hostname cbas2.couchbase.host
2020-06-22T15:42:19.092+00:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized
2020-06-22T15:42:19.098+00:00 INFO DockerTestBase [main] Running 'server-add' for node kv2 (192.168.176.3) with services: [DATA]
2020-06-22T15:42:19.099+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://kv2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data
2020-06-22T15:42:22.141+00:00 INFO DockerTestBase [main+] >> SUCCESS: Server added
2020-06-22T15:42:22.160+00:00 INFO DockerTestBase [main] Running 'server-add' for node cbas1 (192.168.176.4) with services: [ANALYTICS]
2020-06-22T15:42:22.160+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas1.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics
2020-06-22T15:42:25.192+00:00 INFO DockerTestBase [main+] >> SUCCESS: Server added
2020-06-22T15:42:25.206+00:00 INFO DockerTestBase [main] Running 'rebalance'
2020-06-22T15:42:25.206+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase
2020-06-22T15:42:26.431+00:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
2020-06-22T15:42:26.432+00:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete
2020-06-22T15:42:26.437+00:00 INFO DockerTestBase [main] Running 'server-add' for node cbas2 (192.168.176.5) with services: [ANALYTICS]
2020-06-22T15:42:26.437+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas2.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics
2020-06-22T15:42:30.227+00:00 INFO DockerTestBase [main+] >> SUCCESS: Server added
2020-06-22T15:42:30.233+00:00 INFO DockerTestBase [main] Running 'rebalance'
2020-06-22T15:42:30.234+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase
2020-06-22T15:42:37.520+00:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
2020-06-22T15:42:37.520+00:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete
2020-06-22T15:42:37.525+00:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 60s...
2020-06-22T15:42:37.561+00:00 INFO DockerTestBase [main] Creating and loading beer-sample
2020-06-22T15:42:37.562+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/cbdocloader -n 192.168.176.2:8091 -u couchbase -p couchbase -b beer-sample -s 100 /opt/couchbase/samples/beer-sample.zip
2020-06-22T15:42:37.617+00:00 INFO DockerTestBase [main+] >> Warning: Specifying the dataset without the -d/--dataset option is deprecated
2020-06-22T15:42:37.617+00:00 INFO DockerTestBase [main+] >> Warning: -n is deprecated, use -c/--cluster
2020-06-22T15:42:37.617+00:00 INFO DockerTestBase [main+] >> Warning: -s is deprecated, use -m/--bucket-quota
2020-06-22T15:42:47.444+00:00 INFO DockerTestBase [main+] >> Errors occurred during the index creation phase. See logs for details.
2020-06-22T15:42:47.449+00:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.19.0.1:41307]
2020-06-22T15:42:50.021+00:00 INFO DockerTestBase [main] Waiting for all beers to be ingested...
2020-06-22T15:42:50.518+00:00 INFO DockerTestBase [main] ResultSet found : [ {
  "$1" : 0
} ]
2020-06-22T15:42:50.518+00:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : [ {
  "$1" : 0
} ]
2020-06-22T15:42:52.021+00:00 INFO DockerTestBase [main] ResultSet found : [ {
  "$1" : 2936
} ]
2020-06-22T15:42:52.022+00:00 INFO DockerTestBase [main] Failed attempt number: 2. ResultSet found : [ {
  "$1" : 2936
} ]
2020-06-22T15:42:53.139+00:00 INFO DockerTestBase [main] ResultSet found : [ {
  "$1" : 5891
} ]
2020-06-22T15:42:53.139+00:00 INFO DockerTestBase [main] Success
2020-06-22T15:42:53.139+00:00 INFO UpgradeFromAlice600FailoverITD [main] Failing over cbas nodes ...
2020-06-22T15:42:53.139+00:00 INFO DockerTestBase [main] Running 'failover' on [cbas1 (192.168.176.4), cbas2 (192.168.176.5)] nodes
2020-06-22T15:42:53.139+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli failover -c 192.168.176.2:8091 -u couchbase -p couchbase --server-failover cbas1.couchbase.host,cbas2.couchbase.host --hard
2020-06-22T15:42:53.311+00:00 INFO DockerTestBase [main+] >> ERROR: unrecognized arguments: --hard
2020-06-22T15:42:58.328+00:00 INFO UpgradeTestBase [main] Recreating 'cbas2' node (image = null, forceNewIp = false)...
2020-06-22T15:42:58.329+00:00 INFO UpgradeTestBase [ForkJoinPool.commonPool-worker-1] Recreating 'cbas1' node (image = null, forceNewIp = false)...
2020-06-22T15:42:58.329+00:00 INFO DockerTestBase [main] Copying files from docker container 91d28638f657427c81a250d265cf2d49f4b80ee85afd5404643b9fb2e0bc3526: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0012_cbas2_couchbase_enterprise-6.0.0
2020-06-22T15:42:58.330+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Copying files from docker container 26b2aee89c095b437f49e53f4142002e7c7632f9765b40726e24c54b183b0c5e: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0011_cbas1_couchbase_enterprise-6.0.0
2020-06-22T15:42:58.419+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopping docker container: 26b2aee89c095b437f49e53f4142002e7c7632f9765b40726e24c54b183b0c5e
2020-06-22T15:42:58.434+00:00 INFO DockerTestBase [main] Stopping docker container: 91d28638f657427c81a250d265cf2d49f4b80ee85afd5404643b9fb2e0bc3526
2020-06-22T15:42:59.906+00:00 INFO DockerTestBase [main] Stopped docker container: 91d28638f657427c81a250d265cf2d49f4b80ee85afd5404643b9fb2e0bc3526
2020-06-22T15:42:59.906+00:00 INFO DockerTestBase [main] Removing docker container: 91d28638f657427c81a250d265cf2d49f4b80ee85afd5404643b9fb2e0bc3526
2020-06-22T15:42:59.923+00:00 INFO DockerTestBase [main] Removed docker container: 91d28638f657427c81a250d265cf2d49f4b80ee85afd5404643b9fb2e0bc3526
2020-06-22T15:42:59.924+00:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0013_cbas2_thisbuild, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:43:00.755+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Stopped docker container: 26b2aee89c095b437f49e53f4142002e7c7632f9765b40726e24c54b183b0c5e
2020-06-22T15:43:00.756+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removing docker container: 26b2aee89c095b437f49e53f4142002e7c7632f9765b40726e24c54b183b0c5e
2020-06-22T15:43:00.774+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Removed docker container: 26b2aee89c095b437f49e53f4142002e7c7632f9765b40726e24c54b183b0c5e
2020-06-22T15:43:00.776+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0014_cbas1_thisbuild, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:43:02.145+00:00 INFO DockerTestBase [main] Container information for f52b5c1defb1344917df39fccbf577149d002f36dfe7b3188a315e13d448a203: host: cbas2.couchbase.host, ip: 192.168.176.5, 172.19.0.1:41333->192.168.176.5:8095, 172.19.0.1:41337->192.168.176.5:8091
2020-06-22T15:43:03.407+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Container information for e7bbd44c87ab7c2d378fd6c97468d348e739cbc07b90c258f9aeaa0c15718b5d: host: cbas1.couchbase.host, ip: 192.168.176.4, 172.19.0.1:41348->192.168.176.4:8095, 172.19.0.1:41352->192.168.176.4:8091
2020-06-22T15:43:10.444+00:00 INFO DockerTestBase [main] Running 'recovery'
2020-06-22T15:43:10.445+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli recovery -c 192.168.176.2:8091 -u couchbase -p couchbase --server-recovery cbas2.couchbase.host,cbas1.couchbase.host --recovery-type full
2020-06-22T15:43:10.660+00:00 INFO DockerTestBase [main+] >> ERROR: otpNode - invalid node name or node can't be used for delta recovery
2020-06-22T15:43:10.694+00:00 INFO DockerTestBase [main] Running 'rebalance'
2020-06-22T15:43:10.694+00:00 INFO DockerTestBase [main] Executing the command: docker exec fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230 /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase
2020-06-22T15:43:26.736+00:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
2020-06-22T15:43:26.737+00:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete
2020-06-22T15:43:26.742+00:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 60s...
2020-06-22T15:43:26.742+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 60s...
2020-06-22T15:43:26.849+00:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.19.0.1:41352]
2020-06-22T15:43:26.849+00:00 INFO DockerTestBase [main] Waiting for all beers to be ingested...
2020-06-22T15:43:26.851+00:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-1] Initializing KvStoreHttpHelper with nodes [/172.19.0.1:41337]
2020-06-22T15:43:26.852+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Waiting for all beers to be ingested...
2020-06-22T15:43:27.251+00:00 INFO DockerTestBase [main] ResultSet found : [ {
  "$1" : 5891
} ]
2020-06-22T15:43:27.251+00:00 INFO DockerTestBase [main] Success
2020-06-22T15:43:27.420+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] ResultSet found : [ {
  "$1" : 5891
} ]
2020-06-22T15:43:27.420+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-1] Success
2020-06-22T15:43:27.420+00:00 INFO DockerTestBase [main] Running 'failover' on [kv1 (192.168.176.2)] nodes
2020-06-22T15:43:27.420+00:00 INFO DockerTestBase [main] Executing the command: docker exec e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059 /opt/couchbase/bin/couchbase-cli failover -c 192.168.176.3:8091 -u couchbase -p couchbase --server-failover kv1.couchbase.host --hard
2020-06-22T15:43:27.596+00:00 INFO DockerTestBase [main+] >> ERROR: unrecognized arguments: --hard
2020-06-22T15:43:32.610+00:00 INFO UpgradeTestBase [main] Recreating 'kv1' node (image = null, forceNewIp = false)...
2020-06-22T15:43:32.610+00:00 INFO DockerTestBase [main] Copying files from docker container fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0009_kv1_couchbase_enterprise-6.0.0
2020-06-22T15:43:32.759+00:00 INFO DockerTestBase [main] Stopping docker container: fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230
2020-06-22T15:43:34.401+00:00 INFO DockerTestBase [main] Stopped docker container: fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230
2020-06-22T15:43:34.401+00:00 INFO DockerTestBase [main] Removing docker container: fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230
2020-06-22T15:43:34.415+00:00 INFO DockerTestBase [main] Removed docker container: fde9562f0e5ea44e2557ae5a8254104130bfbba86b0a2e4649942a5d41334230
2020-06-22T15:43:34.416+00:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0015_kv1_thisbuild, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:43:36.136+00:00 INFO DockerTestBase [main] Container information for e2066c21978ac5525caf51d3d1f2bc41c50c9f482dc7824eb3ec1b3814176128: host: kv1.couchbase.host, ip: 192.168.176.2, 172.19.0.1:41363->192.168.176.2:8095, 172.19.0.1:41367->192.168.176.2:8091
2020-06-22T15:43:46.322+00:00 INFO DockerTestBase [main] Running 'recovery'
2020-06-22T15:43:46.323+00:00 INFO DockerTestBase [main] Executing the command: docker exec e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059 /opt/couchbase/bin/couchbase-cli recovery -c 192.168.176.3:8091 -u couchbase -p couchbase --server-recovery kv1.couchbase.host --recovery-type full
2020-06-22T15:43:46.532+00:00 INFO DockerTestBase [main+] >> ERROR: otpNode - invalid node name or node can't be used for delta recovery
2020-06-22T15:43:46.550+00:00 INFO DockerTestBase [main] Running 'rebalance'
2020-06-22T15:43:46.550+00:00 INFO DockerTestBase [main] Executing the command: docker exec e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059 /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase
2020-06-22T15:43:56.938+00:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
2020-06-22T15:43:56.939+00:00 INFO DockerTestBase [main+] >> ERROR: Rebalance failed. See logs for detailed reason. You can try again.
2020-06-22T15:43:56.972+00:00 INFO DockerTestBase [main] Running 'failover' on [kv2 (192.168.176.3)] nodes
2020-06-22T15:43:56.972+00:00 INFO DockerTestBase [main] Executing the command: docker exec e2066c21978ac5525caf51d3d1f2bc41c50c9f482dc7824eb3ec1b3814176128 /opt/couchbase/bin/couchbase-cli failover -c 192.168.176.2:8091 -u couchbase -p couchbase --server-failover kv2.couchbase.host --hard
2020-06-22T15:44:02.498+00:00 INFO DockerTestBase [main+] >> SUCCESS: Server failed over
2020-06-22T15:44:07.523+00:00 INFO UpgradeTestBase [main] Recreating 'kv2' node (image = null, forceNewIp = false)...
2020-06-22T15:44:07.524+00:00 INFO DockerTestBase [main] Copying files from docker container e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0010_kv2_couchbase_enterprise-6.0.0
2020-06-22T15:44:07.746+00:00 INFO DockerTestBase [main] Stopping docker container: e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059
2020-06-22T15:44:09.405+00:00 INFO DockerTestBase [main] Stopped docker container: e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059
2020-06-22T15:44:09.406+00:00 INFO DockerTestBase [main] Removing docker container: e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059
2020-06-22T15:44:09.417+00:00 INFO DockerTestBase [main] Removed docker container: e525bdbac9dcec27083cb9f80246c37c68a69430eed6cf1fcb59bd4cdde1b059
2020-06-22T15:44:09.418+00:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600FailoverITD/failoverUpgradeAll/0016_kv2_thisbuild, network = { name = cbas-docker-test-0003-a4c11524-179c-4cce-9133-1859361fc001, id = d187142cbdfb3db9d27ce0fd7ff93b3b08f4442384437cd39547235359d8db1e })
2020-06-22T15:44:10.879+00:00 INFO DockerTestBase [main] Container information for 3fd97daea2bf2603e5960af7cba3223f67db3c235a64bb4152c132a9b2e8d967: host: kv2.couchbase.host, ip: 192.168.176.3, 172.19.0.1:41378->192.168.176.3:8095, 172.19.0.1:41382->192.168.176.3:8091
2020-06-22T15:44:18.092+00:00 INFO DockerTestBase [main] Running 'recovery'
2020-06-22T15:44:18.092+00:00 INFO DockerTestBase [main] Executing the command: docker exec e2066c21978ac5525caf51d3d1f2bc41c50c9f482dc7824eb3ec1b3814176128 /opt/couchbase/bin/couchbase-cli recovery -c 192.168.176.2:8091 -u couchbase -p couchbase --server-recovery kv2.couchbase.host --recovery-type full
2020-06-22T15:44:18.321+00:00 INFO DockerTestBase [main+] >> SUCCESS: Servers recovered
2020-06-22T15:44:18.394+00:00 INFO DockerTestBase [main] Running 'rebalance'
2020-06-22T15:44:18.394+00:00 INFO DockerTestBase [main] Executing the command: docker exec e2066c21978ac5525caf51d3d1f2bc41c50c9f482dc7824eb3ec1b3814176128 /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase
2020-06-22T15:44:29.815+00:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
2020-06-22T15:44:29.816+00:00 INFO DockerTestBase [main+] >> ERROR: Rebalance failed. See logs for detailed reason. You can try again.
2020-06-22T15:44:29.890+00:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 6.6.x' condition for up to 60 seconds...
2020-06-22T15:44:29.891+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Waiting for 'compatibilityVersion promotion to 6.6.x' condition for up to 60 seconds...
2020-06-22T15:44:30.391+00:00 INFO DockerTestBase [pool] assuming 6.0 compatibility on missing 'clusterCompatibility' field
2020-06-22T15:44:30.428+00:00 INFO DockerTestBase [pool] assuming 6.0 compatibility on missing 'clusterCompatibility' field
2020-06-22T15:44:31.445+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c9fbc6d[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@18a03ba1] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:31.795+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:32.457+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1a661e4[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@f618e78] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:32.798+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:33.472+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7dfee015[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@73391002] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:33.801+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:34.490+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6ba1ede[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@87de1a1] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:34.804+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:35.503+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d0a776a[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@53cd8363] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:35.807+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:36.525+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@665460d1[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@689393b3] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:37.538+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1b161048[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@15d1fc82] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:38.552+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6deeca9[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@81903da] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:39.575+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@70ba9924[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@7e7d689b] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:40.592+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.lang.Exception: HTTP error 400:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@301f1b80[Not completed, task = com.couchbase.analytics.servlet.NodeDiagnosticsServlet$$Lambda$360/0x0000000100742c40@3ed8f22b] rejected from org.apache.hyracks.util.MaintainedThreadNameExecutorService@599d319a[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 43]

2020-06-22T15:44:41.596+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.net.SocketException: Connection reset
2020-06-22T15:44:42.600+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:43.604+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: java.net.SocketException: Connection reset
2020-06-22T15:44:44.607+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:45.612+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:46.617+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:47.621+00:00 WARN DockerTestBase [pool] unable to determine cluster compatibility: org.apache.http.NoHttpResponseException: The target server failed to respond
2020-06-22T15:44:48.886+00:00 INFO DockerTestBase [main] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 60s...
2020-06-22T15:44:48.886+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Waiting for NCs == 2 & Cluster state in {ACTIVE} for up to 60s...
2020-06-22T15:44:50.190+00:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.19.0.1:41352]
2020-06-22T15:44:50.191+00:00 INFO DockerTestBase [main] Waiting for all beers to be ingested...
2020-06-22T15:44:50.193+00:00 INFO KvStoreHttpHelper [ForkJoinPool.commonPool-worker-5] Initializing KvStoreHttpHelper with nodes [/172.19.0.1:41337]
2020-06-22T15:44:50.193+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] Waiting for all beers to be ingested...
2020-06-22T15:44:50.910+00:00 INFO DockerTestBase [main] ResultSet found : [ {
  "$1" : 2942
} ]
2020-06-22T15:44:50.910+00:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : [ {
  "$1" : 2942
} ]
2020-06-22T15:44:50.913+00:00 INFO DockerTestBase [ForkJoinPool.commonPool-worker-5] ResultSet found : [ {
  "$1" : 2942
} ]
...