2021-02-02T08:45:49.529-08:00 WARN TestMethodTracer [main] ### upgradeEnsureTxnLog START
2021-02-02T08:45:49.562-08:00 INFO DockerTestBase [main] Creating docker network (name = cbas-docker-test-0005-ce540a7b-e188-49ed-9ddb-d07916ec31c7)
2021-02-02T08:45:49.607-08:00 INFO DockerTestBase [main] Created docker network: { name = cbas-docker-test-0005-ce540a7b-e188-49ed-9ddb-d07916ec31c7, id = d5a8ad4b4ec68eebe3c1d37c75c50e12d30db0b0af4c76fab23dff70516a2eaa }
2021-02-02T08:45:49.608-08:00 INFO UpgradeTestBase [main] Creating kv node ([couchbase:enterprise-6.0.0])...
2021-02-02T08:45:49.608-08:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0001kv to /opt/couchbase/var
2021-02-02T08:45:49.609-08:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0019_kv_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0005-ce540a7b-e188-49ed-9ddb-d07916ec31c7, id = d5a8ad4b4ec68eebe3c1d37c75c50e12d30db0b0af4c76fab23dff70516a2eaa })
2021-02-02T08:45:50.929-08:00 INFO DockerTestBase [main] Container information for 03af676755de9b549a576f78e0dceb4a2896f275b1274a82e074ebb003398939: host: kv.couchbase.host, ip: 192.168.176.2, 172.17.0.1:50041->192.168.176.2:8095, 172.17.0.1:50045->192.168.176.2:8091
2021-02-02T08:46:04.990-08:00 INFO DockerTestBase [main] Running 'node-init' on node kv (192.168.176.2)
2021-02-02T08:46:04.990-08:00 INFO DockerTestBase [main] Executing the command: docker exec 03af676755de9b549a576f78e0dceb4a2896f275b1274a82e074ebb003398939 /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.2:8091 -u couchbase -p couchbase --node-init-hostname kv.couchbase.host
2021-02-02T08:46:05.993-08:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized
2021-02-02T08:46:06.007-08:00 INFO DockerTestBase [main] Running 'cluster-init' on node kv (192.168.176.2) with services: [DATA]
2021-02-02T08:46:06.007-08:00 INFO DockerTestBase [main] Executing the command: docker exec 03af676755de9b549a576f78e0dceb4a2896f275b1274a82e074ebb003398939 /opt/couchbase/bin/couchbase-cli cluster-init -c 127.0.0.1:8091 --cluster-username couchbase --cluster-password couchbase --cluster-ramsize 512 --cluster-analytics-ramsize 1024 --services data
2021-02-02T08:46:06.299-08:00 INFO DockerTestBase [main+] >> SUCCESS: Cluster initialized
2021-02-02T08:46:06.307-08:00 INFO UpgradeTestBase [main] Creating cbas node ([couchbase:enterprise-6.0.0])...
2021-02-02T08:46:06.307-08:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0002cbas to /opt/couchbase/var
2021-02-02T08:46:06.308-08:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0020_cbas_couchbase_enterprise-6.0.0, network = { name = cbas-docker-test-0005-ce540a7b-e188-49ed-9ddb-d07916ec31c7, id = d5a8ad4b4ec68eebe3c1d37c75c50e12d30db0b0af4c76fab23dff70516a2eaa })
2021-02-02T08:46:07.703-08:00 INFO DockerTestBase [main] Container information for 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb: host: cbas.couchbase.host, ip: 192.168.176.3, 172.17.0.1:50056->192.168.176.3:8095, 172.17.0.1:50060->192.168.176.3:8091
2021-02-02T08:46:22.103-08:00 INFO DockerTestBase [main] Running 'node-init' on node cbas (192.168.176.3)
2021-02-02T08:46:22.103-08:00 INFO DockerTestBase [main] Executing the command: docker exec 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.3:8091 -u couchbase -p couchbase --node-init-hostname cbas.couchbase.host
2021-02-02T08:46:23.250-08:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized
2021-02-02T08:46:23.258-08:00 INFO DockerTestBase [main] Running 'server-add' for node cbas (192.168.176.3) with services: [ANALYTICS]
2021-02-02T08:46:23.259-08:00 INFO DockerTestBase [main] Executing the command: docker exec 03af676755de9b549a576f78e0dceb4a2896f275b1274a82e074ebb003398939 /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.2:8091 -u couchbase -p couchbase --server-add http://cbas.couchbase.host --server-add-username couchbase --server-add-password couchbase --services analytics
2021-02-02T08:46:28.095-08:00 INFO DockerTestBase [main+] >> SUCCESS: Server added
2021-02-02T08:46:28.104-08:00 INFO UpgradeTestBase [main] Running 'rebalance'
2021-02-02T08:46:28.104-08:00 INFO DockerTestBase [main] Executing the command: docker exec 03af676755de9b549a576f78e0dceb4a2896f275b1274a82e074ebb003398939 /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.2:8091 -u couchbase -p couchbase
2021-02-02T08:46:32.515-08:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
2021-02-02T08:46:32.515-08:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete
2021-02-02T08:46:32.524-08:00 INFO DockerTestBase [main] Waiting for 'rebalance completion' condition for up to 120 seconds...
2021-02-02T08:46:32.530-08:00 INFO UpgradeTestBase [main] ... rebalance succeeded!
2021-02-02T08:46:32.530-08:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s...
2021-02-02T08:46:38.729-08:00 INFO UpgradeFromAlice600ITD [main] Swap-rebalancing KV node ...
2021-02-02T08:46:38.729-08:00 INFO UpgradeTestBase [main] Creating kv-new node using 'this build' docker image...
2021-02-02T08:46:38.729-08:00 INFO UpgradeTestBase [main] binding /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/binds/0003kvnew to /opt/couchbase/var
2021-02-02T08:46:38.730-08:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0021_kvnew_thisbuild, network = { name = cbas-docker-test-0005-ce540a7b-e188-49ed-9ddb-d07916ec31c7, id = d5a8ad4b4ec68eebe3c1d37c75c50e12d30db0b0af4c76fab23dff70516a2eaa })
2021-02-02T08:46:40.146-08:00 INFO DockerTestBase [main] Container information for 426808136ec7ac4bd9ecd2133ab1e95c325bef285ba26148fdb433fa7edae63c: host: kvnew.couchbase.host, ip: 192.168.176.4, 172.17.0.1:50071->192.168.176.4:8095, 172.17.0.1:50075->192.168.176.4:8091
2021-02-02T08:46:50.211-08:00 INFO DockerTestBase [main] Running 'node-init' on node kvnew (192.168.176.4)
2021-02-02T08:46:50.212-08:00 INFO DockerTestBase [main] Executing the command: docker exec 426808136ec7ac4bd9ecd2133ab1e95c325bef285ba26148fdb433fa7edae63c /opt/couchbase/bin/couchbase-cli node-init -c 192.168.176.4:8091 -u couchbase -p couchbase --node-init-hostname kvnew.couchbase.host
2021-02-02T08:46:50.937-08:00 INFO DockerTestBase [main+] >> SUCCESS: Node initialized
2021-02-02T08:46:50.966-08:00 INFO DockerTestBase [main] Running 'server-add' for node kvnew (192.168.176.4) with services: [DATA]
2021-02-02T08:46:50.967-08:00 INFO DockerTestBase [main] Executing the command: docker exec 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb /opt/couchbase/bin/couchbase-cli server-add -c 192.168.176.3:8091 -u couchbase -p couchbase --server-add http://kvnew.couchbase.host --server-add-username couchbase --server-add-password couchbase --services data
2021-02-02T08:46:54.528-08:00 INFO DockerTestBase [main+] >> SUCCESS: Server added
2021-02-02T08:46:54.537-08:00 INFO DockerTestBase [main] Running 'rebalance-out'
2021-02-02T08:46:54.539-08:00 INFO DockerTestBase [main] Executing the command: docker exec 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb /opt/couchbase/bin/couchbase-cli rebalance -c 192.168.176.3:8091 -u couchbase -p couchbase --server-remove kv.couchbase.host
2021-02-02T08:47:08.712-08:00 INFO DockerTestBase [main+] >> Unable to display progress bar on this os
2021-02-02T08:47:08.712-08:00 INFO DockerTestBase [main+] >> SUCCESS: Rebalance complete
2021-02-02T08:47:08.719-08:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:50075]
2021-02-02T08:47:08.719-08:00 INFO KvStoreHttpHelper [main] installing sample(s): [beer-sample] on 172.17.0.1:50075
2021-02-02T08:47:09.795-08:00 INFO KvStoreHttpHelper [main] Initializing KvStoreHttpHelper with nodes [/172.17.0.1:50060]
2021-02-02T08:47:10.636-08:00 INFO UpgradeFromAlice600ITD [main] Creating magic bucket...
2021-02-02T08:47:10.646-08:00 INFO KvStoreHttpHelper [main] Bucket (magic_bucket) does not exist.
2021-02-02T08:47:10.653-08:00 INFO KvStoreHttpHelper [main] Bucket (magic_bucket) created successfully.
2021-02-02T08:47:10.785-08:00 INFO UpgradeFromAlice600ITD [main] Creating magic dataset...
2021-02-02T08:47:11.069-08:00 INFO UpgradeFromAlice600ITD [main] Creating magic index...
2021-02-02T08:47:11.182-08:00 INFO UpgradeTestBase [main] Connecting link...
2021-02-02T08:47:14.561-08:00 INFO UpgradeFromAlice600ITD [main] Waiting for magic bucket to be ingested...
2021-02-02T08:47:15.007-08:00 INFO DockerTestBase [main] ResultSet found : { "cnt": 0 }
2021-02-02T08:47:15.008-08:00 INFO DockerTestBase [main] Success
2021-02-02T08:47:15.008-08:00 INFO UpgradeFromAlice600ITD [main] Ingesting magic nulls into magic bucket...
2021-02-02T08:47:15.086-08:00 INFO UpgradeFromAlice600ITD [main] Waiting for magic bucket to be ingested...
2021-02-02T08:47:15.213-08:00 INFO DockerTestBase [main] ResultSet found : { "cnt": 0 }
2021-02-02T08:47:15.213-08:00 INFO DockerTestBase [main] Failed attempt number: 1. ResultSet found : { "cnt": 0 }
2021-02-02T08:47:16.279-08:00 INFO DockerTestBase [main] ResultSet found : { "cnt": 3 }
2021-02-02T08:47:16.279-08:00 INFO DockerTestBase [main] Success
2021-02-02T08:47:16.279-08:00 INFO UpgradeFromAlice600ITD [main] Killing node (should ensure txn log) ...
2021-02-02T08:47:16.279-08:00 INFO DockerTestBase [main] Copying files from docker container 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0020_cbas_couchbase_enterprise-6.0.0
2021-02-02T08:47:16.557-08:00 INFO DockerTestBase [main] Stopping docker container: 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb
2021-02-02T08:47:17.158-08:00 INFO DockerTestBase [main] Stopped docker container: 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb
2021-02-02T08:47:17.158-08:00 INFO DockerTestBase [main] Removing docker container: 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb
2021-02-02T08:47:17.167-08:00 INFO DockerTestBase [main] Removed docker container: 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb
2021-02-02T08:47:17.168-08:00 INFO UpgradeFromAlice600ITD [main] Backing up pre-upgrade state ...
2021-02-02T08:47:17.214-08:00 INFO UpgradeFromAlice600ITD [main] Upgrading node ...
2021-02-02T08:47:17.215-08:00 INFO UpgradeTestBase [main] Recreating 'cbas' node (image = null, forceNewIp = false)...
2021-02-02T08:47:17.215-08:00 INFO DockerTestBase [main] Not stopping docker container 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb (not running)
2021-02-02T08:47:17.215-08:00 INFO DockerTestBase [main] Not removing already removed docker container 13f41d445e284fa9bca955e51f566e108d71b93265b42e8372698bd6205f1ebb
2021-02-02T08:47:17.216-08:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0022_cbas_thisbuild, network = { name = cbas-docker-test-0005-ce540a7b-e188-49ed-9ddb-d07916ec31c7, id = d5a8ad4b4ec68eebe3c1d37c75c50e12d30db0b0af4c76fab23dff70516a2eaa })
2021-02-02T08:47:18.649-08:00 INFO DockerTestBase [main] Container information for 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9: host: cbas.couchbase.host, ip: 192.168.176.3, 172.17.0.1:50086->192.168.176.3:8095, 172.17.0.1:50090->192.168.176.3:8091
2021-02-02T08:47:27.681-08:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {ACTIVE,REBALANCE_REQUIRED} for up to 120s...
2021-02-02T08:47:51.114-08:00 INFO DockerTestBase [main] Waiting for 'compatibilityVersion promotion to 7.0.x' condition for up to 120 seconds...
2021-02-02T08:47:51.260-08:00 INFO UpgradeFromAlice600ITD [main] Killing & removing node now that clusterCompat has been updated ...
2021-02-02T08:47:51.260-08:00 INFO DockerTestBase [main] Copying files from docker container 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9: /opt/couchbase/var/lib/couchbase/logs/ to local /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0022_cbas_thisbuild
2021-02-02T08:47:51.590-08:00 INFO DockerTestBase [main] Stopping docker container: 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9
2021-02-02T08:47:52.157-08:00 INFO DockerTestBase [main] Stopped docker container: 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9
2021-02-02T08:47:52.157-08:00 INFO DockerTestBase [main] Removing docker container: 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9
2021-02-02T08:47:52.168-08:00 INFO DockerTestBase [main] Removed docker container: 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9
2021-02-02T08:47:52.168-08:00 INFO UpgradeFromAlice600ITD [main] Restoring pre-upgrade state ...
2021-02-02T08:47:52.235-08:00 INFO UpgradeTestBase [main] Recreating 'cbas' node (image = null, forceNewIp = false)...
2021-02-02T08:47:52.236-08:00 INFO DockerTestBase [main] Not stopping docker container 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9 (not running)
2021-02-02T08:47:52.236-08:00 INFO DockerTestBase [main] Not removing already removed docker container 084ada13d1b347353d7e39e141ac686d757afb8af9ca37612f210dbdd61662f9
2021-02-02T08:47:52.236-08:00 INFO DockerTestBase [main] Creating docker container (test dir = /home/couchbase/jenkins/workspace/cbas-cbcluster-docker-test/analytics/cbas/cbas-test/cbas-docker-test/target/com.couchbase.analytics.test.docker.upgrade.alice.UpgradeFromAlice600ITD/upgradeEnsureTxnLog/0023_cbas_thisbuild, network = { name = cbas-docker-test-0005-ce540a7b-e188-49ed-9ddb-d07916ec31c7, id = d5a8ad4b4ec68eebe3c1d37c75c50e12d30db0b0af4c76fab23dff70516a2eaa })
2021-02-02T08:47:53.723-08:00 INFO DockerTestBase [main] Container information for 7597e66cf60435155c0e95634e73af3ee8e001079d886741fc5f1f430843eedc: host: cbas.couchbase.host, ip: 192.168.176.3, 172.17.0.1:50101->192.168.176.3:8095, 172.17.0.1:50105->192.168.176.3:8091
2021-02-02T08:48:02.811-08:00 INFO UpgradeFromAlice600ITD [main] Waiting for REBALANCE_REQUIRED ...
2021-02-02T08:48:02.812-08:00 INFO DockerTestBase [main] Waiting for NCs == 1 & Cluster state in {REBALANCE_REQUIRED} for up to 120s...
2021-02-02T08:50:02.813-08:00 WARN DockerTestBase [main] Timed out waiting for condition: NCs == 1 (last count was -1) & Cluster [REBALANCE_REQUIRED] (last state was null), last error was java.net.SocketException: Connection reset, attemptCount: 120
java.net.SocketException: Connection reset