Started by user Bryan McCoid [EnvInject] - Loading node environment variables. Building remotely on slv-sc2402-32g-12c (volume_test_collections) in workspace /data/workspace/temp_rebalance_even_win_MB-45061 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_rebalance_even_win_MB-45061 # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision ee12872bd8a6031b563ebf650b0cf4063dd4ee17 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f ee12872bd8a6031b563ebf650b0cf4063dd4ee17 > /usr/bin/git rev-list ee12872bd8a6031b563ebf650b0cf4063dd4ee17 # timeout=10 [temp_rebalance_even_win_MB-45061] $ /bin/sh -xe /tmp/jenkins2805578924308151708.sh + echo '[global] username:Administrator password:Membase123 port:8091 index_port:9102 n1ql_port:8093 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 [_1] ip:172.23.136.191 services:kv [_2] ip:172.23.136.192 services:kv [_3] ip:172.23.136.193 services:kv [_5] ip:172.23.136.194 services:kv [_4] ip:172.23.136.195 services:kv' [temp_rebalance_even_win_MB-45061] $ /bin/sh -xe /tmp/jenkins3569187032392660892.sh + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From https://github.com/couchbaselabs/TAF * branch master -> FETCH_HEAD Already up-to-date. + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + guides/gradlew check -s Starting a Gradle Daemon, 47 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 > Task :compileJava Note: /data/workspace/temp_rebalance_even_win_MB-45061/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :processResources > Task :classes > Task :compileTestJava NO-SOURCE > Task :processTestResources NO-SOURCE > Task :testClasses UP-TO-DATE > Task :test NO-SOURCE > Task :check UP-TO-DATE Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 5s 2 actionable tasks: 2 executed + guides/gradlew -v ------------------------------------------------------------ Gradle 6.2.2 ------------------------------------------------------------ Build time: 2020-03-04 08:49:31 UTC Revision: 7d0bf6dcb46c143bcc3b7a0fa40a8e5ca28e5856 Kotlin: 1.3.61 Groovy: 2.5.8 Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 JVM: 1.8.0_242 (Oracle Corporation 25.242-b08) OS: Linux 3.10.0-1062.18.1.el7.x86_64 amd64 + git fetch http://review.couchbase.org/TAF refs/changes/35/149035/3 From http://review.couchbase.org/TAF * branch refs/changes/35/149035/3 -> FETCH_HEAD + git cherry-pick FETCH_HEAD [master 8ef73aa] Increase http timeout for windows runs Author: Balakumaran G 2 files changed, 6 insertions(+), 6 deletions(-) + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_failover.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'GROUP': 'P0_failover_and_recovery', 'change_ephemeral_purge_age_and_interval': 'False', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'num_nodes': 5, 'rerun': 'False', 'spec': 'collections_failover'} Only cases in GROUPs 'P0_failover_and_recovery' will be executed Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_1', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 1, 'scrape_interval': '5'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:45:20,088 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:45:20,898 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:45:52,525 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'665b3dbfa1dc93a4b99a4b7bf7a03cc4', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:45:52,585 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:45:52,586 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274751303L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:51.303Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 06b482654e6afeb68c8fa6c7be4457f5"} 2021-04-12 17:45:52,588 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274721298L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:21.298Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 06b482654e6afeb68c8fa6c7be4457f5"} 2021-04-12 17:45:52,588 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274721285L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:45:21.285Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:45:52,589 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.136.194', u'tstamp': 1618274694595L, u'shortText': u'web start ok', u'serverTime': u'2021-04-12T17:44:54.595Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.136.194\'. Version: "7.0.0-4902-enterprise".'} 2021-04-12 17:45:52,592 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274571878L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:42:51.878Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 24.284652 secs ago)'} 2021-04-12 17:45:52,592 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274562613L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:42:42.613Z', u'text': u'Rebalance exited with reason {{badmatch,\n {leader_activities_error,\n {default,rebalance},\n {no_quorum,\n [{required_quorum,majority},\n {leases,\n [\'ns_1@172.23.136.191\',\n \'ns_1@172.23.136.192\',\n \'ns_1@172.23.136.195\']}]}}},\n [{ns_rebalancer,rebalance,5,\n [{file,"src/ns_rebalancer.erl"},{line,490}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,249}]}]}.\nRebalance Operation Id = 0a5b60f72a421abf72845ebb84265dc2'} 2021-04-12 17:45:52,592 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274547602L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:42:27.602Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 0a5b60f72a421abf72845ebb84265dc2"} 2021-04-12 17:45:52,594 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274540248L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:42:20.248Z', u'text': u'Rebalance exited with reason {{badmatch,\n {leader_activities_error,\n {default,rebalance},\n {no_quorum,\n [{required_quorum,majority},\n {leases,\n [\'ns_1@172.23.136.191\',\n \'ns_1@172.23.136.192\',\n \'ns_1@172.23.136.195\']}]}}},\n [{ns_rebalancer,rebalance,5,\n [{file,"src/ns_rebalancer.erl"},{line,490}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,249}]}]}.\nRebalance Operation Id = 6815eb13db1a565f57c2b689da56c881'} 2021-04-12 17:45:52,595 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274525229L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:42:05.229Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 6815eb13db1a565f57c2b689da56c881"} 2021-04-12 17:45:52,595 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274525221L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:42:05.221Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'665b3dbfa1dc93a4b99a4b7bf7a03cc4', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'665b3dbfa1dc93a4b99a4b7bf7a03cc4', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 37.163s FAILED (errors=1) During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 1 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_1 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_2', 'nodes_init': '5', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 2} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:45:57,398 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:45:58,141 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:46:29,703 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'93ed2a60a15cba23da96624f18d5620a', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:46:29,739 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:46:29,739 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274788421L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:46:28.421Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 62f11008fa43482c58eadded3acb46be"} 2021-04-12 17:46:29,740 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274758412L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:58.412Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 62f11008fa43482c58eadded3acb46be"} 2021-04-12 17:46:29,743 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274758402L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:45:58.402Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:46:29,743 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274751303L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:51.303Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 06b482654e6afeb68c8fa6c7be4457f5"} 2021-04-12 17:46:29,744 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274721298L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:21.298Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 06b482654e6afeb68c8fa6c7be4457f5"} 2021-04-12 17:46:29,746 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274721285L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:45:21.285Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:46:29,746 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.136.194', u'tstamp': 1618274694595L, u'shortText': u'web start ok', u'serverTime': u'2021-04-12T17:44:54.595Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.136.194\'. Version: "7.0.0-4902-enterprise".'} 2021-04-12 17:46:29,746 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274571878L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:42:51.878Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 24.284652 secs ago)'} 2021-04-12 17:46:29,747 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274562613L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:42:42.613Z', u'text': u'Rebalance exited with reason {{badmatch,\n {leader_activities_error,\n {default,rebalance},\n {no_quorum,\n [{required_quorum,majority},\n {leases,\n [\'ns_1@172.23.136.191\',\n \'ns_1@172.23.136.192\',\n \'ns_1@172.23.136.195\']}]}}},\n [{ns_rebalancer,rebalance,5,\n [{file,"src/ns_rebalancer.erl"},{line,490}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,249}]}]}.\nRebalance Operation Id = 0a5b60f72a421abf72845ebb84265dc2'} 2021-04-12 17:46:29,749 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274547602L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:42:27.602Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 0a5b60f72a421abf72845ebb84265dc2"} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'93ed2a60a15cba23da96624f18d5620a', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'93ed2a60a15cba23da96624f18d5620a', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 37.123s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 2 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_2 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'after', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_3', 'nodes_init': '5', 'rebalance_moves_per_node': '64', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 3, 'scrape_interval': '2'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:46:34,134 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:46:34,822 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:47:06,134 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'31151b3aa4dc7116ecfe6e23eb404239', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:47:06,167 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:47:06,167 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274825081L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:05.081Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = c00fd563a8e549ed7a23a24ab9ae3c59"} 2021-04-12 17:47:06,168 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274811893L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:46:51.893Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 16.814035 secs ago)'} 2021-04-12 17:47:06,168 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274795088L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:46:35.088Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = c00fd563a8e549ed7a23a24ab9ae3c59"} 2021-04-12 17:47:06,170 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274788421L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:46:28.421Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 62f11008fa43482c58eadded3acb46be"} 2021-04-12 17:47:06,170 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274758412L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:58.412Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 62f11008fa43482c58eadded3acb46be"} 2021-04-12 17:47:06,171 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274758402L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:45:58.402Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:47:06,171 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274751303L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:51.303Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 06b482654e6afeb68c8fa6c7be4457f5"} 2021-04-12 17:47:06,171 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274721298L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:21.298Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 06b482654e6afeb68c8fa6c7be4457f5"} 2021-04-12 17:47:06,173 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274721285L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:45:21.285Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:47:06,173 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.136.194', u'tstamp': 1618274694595L, u'shortText': u'web start ok', u'serverTime': u'2021-04-12T17:44:54.595Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.136.194\'. Version: "7.0.0-4902-enterprise".'} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'31151b3aa4dc7116ecfe6e23eb404239', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'31151b3aa4dc7116ecfe6e23eb404239', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 36.407s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 3 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_3 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_4', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 4, 'scrape_interval': '5'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:47:10,450 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:47:11,121 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:47:42,595 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f0a7b3ddadc6ab7d188a59e0725f9265', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:47:42,628 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:47:42,628 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274861453L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:41.453Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = b56b18d967a28563265e0cd363f0a6bb"} 2021-04-12 17:47:42,628 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274831456L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:11.456Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = b56b18d967a28563265e0cd363f0a6bb"} 2021-04-12 17:47:42,630 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274831445L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:47:11.445Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:47:42,630 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274825081L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:05.081Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = c00fd563a8e549ed7a23a24ab9ae3c59"} 2021-04-12 17:47:42,631 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274811893L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:46:51.893Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 16.814035 secs ago)'} 2021-04-12 17:47:42,631 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274795088L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:46:35.088Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = c00fd563a8e549ed7a23a24ab9ae3c59"} 2021-04-12 17:47:42,631 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274788421L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:46:28.421Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 62f11008fa43482c58eadded3acb46be"} 2021-04-12 17:47:42,631 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274758412L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:58.412Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 62f11008fa43482c58eadded3acb46be"} 2021-04-12 17:47:42,632 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274758402L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:45:58.402Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} Traceback (most recent call last): 2021-04-12 17:47:42,632 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274751303L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:45:51.303Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 06b482654e6afeb68c8fa6c7be4457f5"} File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f0a7b3ddadc6ab7d188a59e0725f9265', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp During the test, super(CollectionsRebalance, self).setUp() Remote Connections: 6, Disconnections: 6 File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) SDK Connections: 0, Disconnections: 0 AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f0a7b3ddadc6ab7d188a59e0725f9265', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 36.444s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 4 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_4 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_5', 'nodes_init': '5', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 5} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:47:47,059 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:47:47,762 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:48:19,259 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f72978adc6734f1c57f25922470c8727', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:48:19,290 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:48:19,290 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274898253L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:18.253Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 152126251cad3678f2f2722e98c603fa"} 2021-04-12 17:48:19,292 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274871896L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:47:51.896Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 3.64663 secs ago)'} 2021-04-12 17:48:19,292 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274868259L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:48.259Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 152126251cad3678f2f2722e98c603fa"} 2021-04-12 17:48:19,292 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274861453L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:41.453Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = b56b18d967a28563265e0cd363f0a6bb"} 2021-04-12 17:48:19,293 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274831456L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:11.456Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = b56b18d967a28563265e0cd363f0a6bb"} 2021-04-12 17:48:19,293 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274831445L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:47:11.445Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:48:19,295 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274825081L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:05.081Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = c00fd563a8e549ed7a23a24ab9ae3c59"} 2021-04-12 17:48:19,295 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274811893L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:46:51.893Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 16.814035 secs ago)'} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f72978adc6734f1c57f25922470c8727', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f72978adc6734f1c57f25922470c8727', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 36.647s FAILED (errors=1) 2021-04-12 17:48:19,296 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274795088L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:46:35.088Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = c00fd563a8e549ed7a23a24ab9ae3c59"} 2021-04-12 17:48:19,296 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274788421L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:46:28.421Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 62f11008fa43482c58eadded3acb46be"} During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 5 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_5 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'after', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_6', 'nodes_init': '5', 'rebalance_moves_per_node': '64', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 6, 'scrape_interval': '2'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:48:23,471 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:48:24,147 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-12 17:48:55,471 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ec57bbec5eb8601140ce8644c905e577', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:48:55,503 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:48:55,503 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274934466L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:54.466Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = fcc214ac50b3b3134ecb58c3ddd74641"} 2021-04-12 17:48:55,503 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274904482L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:24.482Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = fcc214ac50b3b3134ecb58c3ddd74641"} 2021-04-12 17:48:55,503 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274904463L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:48:24.463Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:48:55,503 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274898253L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:18.253Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 152126251cad3678f2f2722e98c603fa"} 2021-04-12 17:48:55,505 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274871896L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:47:51.896Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 3.64663 secs ago)'} 2021-04-12 17:48:55,505 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274868259L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:48.259Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 152126251cad3678f2f2722e98c603fa"} 2021-04-12 17:48:55,506 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274861453L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:41.453Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = b56b18d967a28563265e0cd363f0a6bb"} 2021-04-12 17:48:55,506 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274831456L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:11.456Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = b56b18d967a28563265e0cd363f0a6bb"} 2021-04-12 17:48:55,506 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274831445L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:47:11.445Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:48:55,506 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274825081L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:05.081Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = c00fd563a8e549ed7a23a24ab9ae3c59"} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ec57bbec5eb8601140ce8644c905e577', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'ec57bbec5eb8601140ce8644c905e577', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 36.197s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 6 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_6 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_7 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_7', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 7, 'scrape_interval': '5'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:49:00,470 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #7 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 17:49:01,168 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #7 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 17:49:32,496 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'aaa3fff224a25b3e0069ffe244c9d1e4', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:49:32,526 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:49:32,528 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274971478L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:49:31.478Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 2bfaa41b46facfe876825fd81b95c1d7"} 2021-04-12 17:49:32,529 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274941488L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:49:01.488Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 2bfaa41b46facfe876825fd81b95c1d7"} 2021-04-12 17:49:32,529 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274941480L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:49:01.480Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:49:32,529 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274934466L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:54.466Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = fcc214ac50b3b3134ecb58c3ddd74641"} 2021-04-12 17:49:32,529 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274904482L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:24.482Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = fcc214ac50b3b3134ecb58c3ddd74641"} 2021-04-12 17:49:32,529 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274904463L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:48:24.463Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:49:32,530 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274898253L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:18.253Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 152126251cad3678f2f2722e98c603fa"} 2021-04-12 17:49:32,530 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274871896L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:47:51.896Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 3.64663 secs ago)'} 2021-04-12 17:49:32,532 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274868259L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:48.259Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 152126251cad3678f2f2722e98c603fa"} 2021-04-12 17:49:32,532 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274861453L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:47:41.453Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = b56b18d967a28563265e0cd363f0a6bb"} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'aaa3fff224a25b3e0069ffe244c9d1e4', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'aaa3fff224a25b3e0069ffe244c9d1e4', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 37.008s FAILED (errors=1) During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 7 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_7 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_8 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_8', 'nodes_init': '5', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 8} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:49:36,691 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 17:49:37,368 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #8 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 17:50:08,641 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'24048602abfd926a276726ebe0c05090', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 17:50:08,671 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 17:50:08,671 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618275007698L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:50:07.698Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 37230217c5ae4fd0b4315a540be63743"} 2021-04-12 17:50:08,673 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.136.193', u'tstamp': 1618274999077L, u'shortText': u'web start ok', u'serverTime': u'2021-04-12T17:49:59.077Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.136.193\'. Version: "7.0.0-4902-enterprise".'} 2021-04-12 17:50:08,673 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274991904L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:49:51.904Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\'] (repeated 1 times, last seen 14.212823 secs ago)'} 2021-04-12 17:50:08,673 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274977700L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:49:37.700Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 37230217c5ae4fd0b4315a540be63743"} 2021-04-12 17:50:08,674 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274971478L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:49:31.478Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = 2bfaa41b46facfe876825fd81b95c1d7"} 2021-04-12 17:50:08,674 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274941488L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:49:01.488Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = 2bfaa41b46facfe876825fd81b95c1d7"} 2021-04-12 17:50:08,674 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274941480L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:49:01.480Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 17:50:08,674 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274934466L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:54.466Z', u'text': u"Rebalance exited with reason {prepare_rebalance_failed,\n {error,\n {failed_nodes,\n [{'ns_1@172.23.136.193',{error,timeout}}]}}}.\nRebalance Operation Id = fcc214ac50b3b3134ecb58c3ddd74641"} 2021-04-12 17:50:08,674 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274904482L, u'shortText': u'message', u'serverTime': u'2021-04-12T17:48:24.482Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193'], Failed over and being ejected nodes = ['ns_1@172.23.136.195']; no delta recovery nodes; Operation Id = fcc214ac50b3b3134ecb58c3ddd74641"} 2021-04-12 17:50:08,676 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618274904463L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T17:48:24.463Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'24048602abfd926a276726ebe0c05090', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'24048602abfd926a276726ebe0c05090', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 36.129s FAILED (errors=1) During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 8 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_8 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_9 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'after', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_9', 'nodes_init': '5', 'rebalance_moves_per_node': '64', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 9, 'scrape_interval': '2'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 17:50:12,780 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 17:50:13,437 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 17:50:41,102 | test | INFO | MainThread | [rest_client:monitorRebalance:1446] Rebalance done. Taken 27.378000021 seconds to complete 2021-04-12 17:50:41,112 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0x85957669, 0.0.0.0/0.0.0.0:52120] timeout=300.0>) 2021-04-12 17:50:46,124 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x67c35a61, 0.0.0.0/0.0.0.0:56966] timeout=300.0>) 2021-04-12 17:50:48,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x076b9ea7, 0.0.0.0/0.0.0.0:39217] timeout=300.0>) 2021-04-12 17:50:50,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xb3e3a6fe, 0.0.0.0/0.0.0.0:34466] timeout=300.0>) 2021-04-12 17:50:52,167 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xbdedaa4b, 0.0.0.0/0.0.0.0:46810] timeout=300.0>) 2021-04-12 17:50:54,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x8645385f, 0.0.0.0/0.0.0.0:49046] timeout=300.0>) 2021-04-12 17:50:56,197 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x77083d9c, 0.0.0.0/0.0.0.0:56732] timeout=300.0>) 2021-04-12 17:50:58,210 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x3f59a1a0, 0.0.0.0/0.0.0.0:36057] timeout=300.0>) 2021-04-12 17:51:00,224 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x6493539b, 0.0.0.0/0.0.0.0:47062] timeout=300.0>) 2021-04-12 17:51:02,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x04746dfd, 0.0.0.0/0.0.0.0:45069] timeout=300.0>) 2021-04-12 17:51:04,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x517ee49c, 0.0.0.0/0.0.0.0:50657] timeout=300.0>) 2021-04-12 17:51:06,267 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0xd51d3b86, 0.0.0.0/0.0.0.0:37106] timeout=300.0>) 2021-04-12 17:51:08,282 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x290ae218, 0.0.0.0/0.0.0.0:54674] timeout=300.0>) 2021-04-12 17:51:10,302 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xb05c4d13, 0.0.0.0/0.0.0.0:56709] timeout=300.0>) 2021-04-12 17:51:12,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x8c629bc1, 0.0.0.0/0.0.0.0:36171] timeout=300.0>) 2021-04-12 17:51:14,328 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0xfae9c774, 0.0.0.0/0.0.0.0:48618] timeout=300.0>) 2021-04-12 17:51:16,344 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x17e0e1d7, 0.0.0.0/0.0.0.0:44980] timeout=300.0>) 2021-04-12 17:51:18,358 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0x80f70ded, 0.0.0.0/0.0.0.0:56942] timeout=300.0>) 2021-04-12 17:51:20,371 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x37c378ba, 0.0.0.0/0.0.0.0:33698] timeout=300.0>) 2021-04-12 17:51:22,385 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0x1dccf1e1, 0.0.0.0/0.0.0.0:51301] timeout=300.0>) 2021-04-12 17:51:24,400 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0x8969b1bf, 0.0.0.0/0.0.0.0:32784] timeout=300.0>) 2021-04-12 17:51:26,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x6ff0a9cc, 0.0.0.0/0.0.0.0:39678] timeout=300.0>) 2021-04-12 17:51:28,430 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0x5ba33d46, 0.0.0.0/0.0.0.0:46117] timeout=300.0>) 2021-04-12 17:51:30,443 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x24 type=client open_count=1 channel=[id: 0x652cc376, 0.0.0.0/0.0.0.0:57565] timeout=300.0>) 2021-04-12 17:51:32,459 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x25 type=client open_count=1 channel=[id: 0xdb518a0b, 0.0.0.0/0.0.0.0:48847] timeout=300.0>) 2021-04-12 17:51:34,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x26 type=client open_count=1 channel=[id: 0xa5fa8565, 0.0.0.0/0.0.0.0:43983] timeout=300.0>) 2021-04-12 17:51:36,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x27 type=client open_count=1 channel=[id: 0x5aef45bc, 0.0.0.0/0.0.0.0:60634] timeout=300.0>) 2021-04-12 17:51:38,497 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x28 type=client open_count=1 channel=[id: 0x7e9b7392, 0.0.0.0/0.0.0.0:55338] timeout=300.0>) 2021-04-12 17:51:40,509 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x29 type=client open_count=1 channel=[id: 0x40b6d2cc, 0.0.0.0/0.0.0.0:36115] timeout=300.0>) 2021-04-12 17:51:42,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x2a type=client open_count=1 channel=[id: 0x4cf9eb60, 0.0.0.0/0.0.0.0:57991] timeout=300.0>) 2021-04-12 17:51:44,536 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x2b type=client open_count=1 channel=[id: 0xe959c787, 0.0.0.0/0.0.0.0:53237] timeout=300.0>) 2021-04-12 17:51:46,548 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x2c type=client open_count=1 channel=[id: 0x60630ac9, 0.0.0.0/0.0.0.0:55062] timeout=300.0>) 2021-04-12 17:51:48,562 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x2d type=client open_count=1 channel=[id: 0xf1c8ca65, 0.0.0.0/0.0.0.0:39248] timeout=300.0>) 2021-04-12 17:51:50,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x2e type=client open_count=1 channel=[id: 0xba9b56da, 0.0.0.0/0.0.0.0:33992] timeout=300.0>) 2021-04-12 17:51:52,586 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x2f type=client open_count=1 channel=[id: 0x32ac5fa2, 0.0.0.0/0.0.0.0:57931] timeout=300.0>) 2021-04-12 17:51:54,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x30 type=client open_count=1 channel=[id: 0x13f2d560, 0.0.0.0/0.0.0.0:46612] timeout=300.0>) 2021-04-12 17:51:56,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x31 type=client open_count=1 channel=[id: 0x69247c0d, 0.0.0.0/0.0.0.0:56843] timeout=300.0>) 2021-04-12 17:51:58,628 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x32 type=client open_count=1 channel=[id: 0xad148de3, 0.0.0.0/0.0.0.0:49530] timeout=300.0>) 2021-04-12 17:52:00,641 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x33 type=client open_count=1 channel=[id: 0x4b6df120, 0.0.0.0/0.0.0.0:57986] timeout=300.0>) 2021-04-12 17:52:02,654 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x34 type=client open_count=1 channel=[id: 0xd498a5d3, 0.0.0.0/0.0.0.0:37652] timeout=300.0>) 2021-04-12 17:52:04,667 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x35 type=client open_count=1 channel=[id: 0x2a6a0650, 0.0.0.0/0.0.0.0:43767] timeout=300.0>) 2021-04-12 17:52:06,681 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x36 type=client open_count=1 channel=[id: 0x031f8c01, 0.0.0.0/0.0.0.0:42107] timeout=300.0>) 2021-04-12 17:52:08,694 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x37 type=client open_count=1 channel=[id: 0x63f5d3a6, 0.0.0.0/0.0.0.0:48700] timeout=300.0>) 2021-04-12 17:52:10,706 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x38 type=client open_count=1 channel=[id: 0x322fe750, 0.0.0.0/0.0.0.0:53700] timeout=300.0>) 2021-04-12 17:52:12,719 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x39 type=client open_count=1 channel=[id: 0xc2c4318f, 0.0.0.0/0.0.0.0:59161] timeout=300.0>) 2021-04-12 17:52:14,732 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x3a type=client open_count=1 channel=[id: 0xaf783cac, 0.0.0.0/0.0.0.0:36596] timeout=300.0>) 2021-04-12 17:52:16,744 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x3b type=client open_count=1 channel=[id: 0x804651ef, 0.0.0.0/0.0.0.0:59316] timeout=300.0>) 2021-04-12 17:52:18,757 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x3c type=client open_count=1 channel=[id: 0x72b08039, 0.0.0.0/0.0.0.0:50119] timeout=300.0>) 2021-04-12 17:52:20,770 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x3d type=client open_count=1 channel=[id: 0x515c567b, 0.0.0.0/0.0.0.0:45075] timeout=300.0>) 2021-04-12 17:52:22,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x3e type=client open_count=1 channel=[id: 0xa76705cf, 0.0.0.0/0.0.0.0:53492] timeout=300.0>) 2021-04-12 17:52:24,796 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x3f type=client open_count=1 channel=[id: 0xa8487847, 0.0.0.0/0.0.0.0:36552] timeout=300.0>) 2021-04-12 17:52:26,809 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x40 type=client open_count=1 channel=[id: 0x20c784a6, 0.0.0.0/0.0.0.0:57115] timeout=300.0>) 2021-04-12 17:52:28,822 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x41 type=client open_count=1 channel=[id: 0xeaf4949e, 0.0.0.0/0.0.0.0:53229] timeout=300.0>) 2021-04-12 17:52:30,835 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x42 type=client open_count=1 channel=[id: 0x3322224d, 0.0.0.0/0.0.0.0:57888] timeout=300.0>) 2021-04-12 17:52:32,848 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x43 type=client open_count=1 channel=[id: 0x9d433311, 0.0.0.0/0.0.0.0:45365] timeout=300.0>) 2021-04-12 17:52:34,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x44 type=client open_count=1 channel=[id: 0x90e3b8e1, 0.0.0.0/0.0.0.0:40551] timeout=300.0>) 2021-04-12 17:52:36,871 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x45 type=client open_count=1 channel=[id: 0x992258d2, 0.0.0.0/0.0.0.0:58031] timeout=300.0>) 2021-04-12 17:52:38,885 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x46 type=client open_count=1 channel=[id: 0x560fcc7a, 0.0.0.0/0.0.0.0:40084] timeout=300.0>) 2021-04-12 17:52:40,901 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x47 type=client open_count=1 channel=[id: 0x22a7bd42, 0.0.0.0/0.0.0.0:56777] timeout=300.0>) 2021-04-12 17:52:42,911 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x48 type=client open_count=1 channel=[id: 0x295fe3e8, 0.0.0.0/0.0.0.0:47759] timeout=300.0>) 2021-04-12 17:52:44,924 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x49 type=client open_count=1 channel=[id: 0x8ae333c7, 0.0.0.0/0.0.0.0:44784] timeout=300.0>) 2021-04-12 17:52:46,937 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x4a type=client open_count=1 channel=[id: 0xfc33389e, 0.0.0.0/0.0.0.0:50685] timeout=300.0>) 2021-04-12 17:52:48,947 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x4b type=client open_count=1 channel=[id: 0xd18d5ee3, 0.0.0.0/0.0.0.0:38484] timeout=300.0>) 2021-04-12 17:52:50,960 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x4c type=client open_count=1 channel=[id: 0x65b4f1ee, 0.0.0.0/0.0.0.0:58427] timeout=300.0>) 2021-04-12 17:52:52,973 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x4d type=client open_count=1 channel=[id: 0x49860082, 0.0.0.0/0.0.0.0:47463] timeout=300.0>) 2021-04-12 17:52:54,984 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x4e type=client open_count=1 channel=[id: 0x394e5f76, 0.0.0.0/0.0.0.0:57284] timeout=300.0>) 2021-04-12 17:52:56,997 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x4f type=client open_count=1 channel=[id: 0x3f767b92, 0.0.0.0/0.0.0.0:54384] timeout=300.0>) 2021-04-12 17:52:59,012 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x50 type=client open_count=1 channel=[id: 0xdfdde6da, 0.0.0.0/0.0.0.0:56484] timeout=300.0>) 2021-04-12 17:53:01,023 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x51 type=client open_count=1 channel=[id: 0x6a225ad8, 0.0.0.0/0.0.0.0:58503] timeout=300.0>) 2021-04-12 17:53:03,035 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x52 type=client open_count=1 channel=[id: 0x2d8298b3, 0.0.0.0/0.0.0.0:53832] timeout=300.0>) 2021-04-12 17:53:05,046 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x53 type=client open_count=1 channel=[id: 0xfa38f101, 0.0.0.0/0.0.0.0:58275] timeout=300.0>) 2021-04-12 17:53:07,059 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x54 type=client open_count=1 channel=[id: 0x2657a0c6, 0.0.0.0/0.0.0.0:37425] timeout=300.0>) 2021-04-12 17:53:09,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x55 type=client open_count=1 channel=[id: 0x411865f7, 0.0.0.0/0.0.0.0:60039] timeout=300.0>) 2021-04-12 17:53:11,085 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x56 type=client open_count=1 channel=[id: 0x127a0578, 0.0.0.0/0.0.0.0:35719] timeout=300.0>) 2021-04-12 17:53:13,101 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x57 type=client open_count=1 channel=[id: 0x7c23c576, 0.0.0.0/0.0.0.0:53700] timeout=300.0>) 2021-04-12 17:53:15,112 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x58 type=client open_count=1 channel=[id: 0xc8666a81, 0.0.0.0/0.0.0.0:58316] timeout=300.0>) 2021-04-12 17:53:17,125 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x59 type=client open_count=1 channel=[id: 0x9252e2de, 0.0.0.0/0.0.0.0:58657] timeout=300.0>) 2021-04-12 17:53:19,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x5a type=client open_count=1 channel=[id: 0x8b63cae0, 0.0.0.0/0.0.0.0:42724] timeout=300.0>) 2021-04-12 17:53:21,153 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x5b type=client open_count=1 channel=[id: 0xc4bc264a, 0.0.0.0/0.0.0.0:59343] timeout=300.0>) 2021-04-12 17:53:23,164 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x5c type=client open_count=1 channel=[id: 0xcd8b5e0e, 0.0.0.0/0.0.0.0:43864] timeout=300.0>) 2021-04-12 17:53:25,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x5d type=client open_count=1 channel=[id: 0x164596e0, 0.0.0.0/0.0.0.0:34342] timeout=300.0>) 2021-04-12 17:53:27,190 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x5e type=client open_count=1 channel=[id: 0x914d1c75, 0.0.0.0/0.0.0.0:43916] timeout=300.0>) 2021-04-12 17:53:29,203 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x5f type=client open_count=1 channel=[id: 0xb3abb66a, 0.0.0.0/0.0.0.0:47123] timeout=300.0>) 2021-04-12 17:53:31,214 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x60 type=client open_count=1 channel=[id: 0xc0539723, 0.0.0.0/0.0.0.0:44440] timeout=300.0>) 2021-04-12 17:53:33,227 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x61 type=client open_count=1 channel=[id: 0xf0e20226, 0.0.0.0/0.0.0.0:34262] timeout=300.0>) 2021-04-12 17:53:35,242 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x62 type=client open_count=1 channel=[id: 0x3ca116ba, 0.0.0.0/0.0.0.0:44651] timeout=300.0>) 2021-04-12 17:53:37,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x63 type=client open_count=1 channel=[id: 0x9cba2cd9, 0.0.0.0/0.0.0.0:54146] timeout=300.0>) 2021-04-12 17:53:39,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x64 type=client open_count=1 channel=[id: 0x8feb35ab, 0.0.0.0/0.0.0.0:55984] timeout=300.0>) 2021-04-12 17:53:41,282 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0x65 type=client open_count=1 channel=[id: 0xe4fc6629, 0.0.0.0/0.0.0.0:34508] timeout=300.0>) 2021-04-12 17:53:43,295 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused 2021-04-12 17:54:18,973 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x66 type=client open_count=1 channel=[id: 0x39ec83b8, 0.0.0.0/0.0.0.0:42343] timeout=300.0>) 2021-04-12 17:54:54,926 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x67 type=client open_count=1 channel=[id: 0xac4ae5d0, 0.0.0.0/0.0.0.0:34284] timeout=300.0>) 2021-04-12 17:54:56,937 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x68 type=client open_count=1 channel=[id: 0xa8bf9fa3, 0.0.0.0/0.0.0.0:44102] timeout=300.0>) 2021-04-12 17:54:58,950 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x69 type=client open_count=1 channel=[id: 0x6641214b, 0.0.0.0/0.0.0.0:46687] timeout=300.0>) 2021-04-12 17:55:00,967 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x6a type=client open_count=1 channel=[id: 0x46eb9e59, 0.0.0.0/0.0.0.0:43540] timeout=300.0>) 2021-04-12 17:55:02,977 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x6b type=client open_count=1 channel=[id: 0x94c69147, 0.0.0.0/0.0.0.0:44092] timeout=300.0>) 2021-04-12 17:55:04,989 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x6c type=client open_count=1 channel=[id: 0xe39624ef, 0.0.0.0/0.0.0.0:50763] timeout=300.0>) 2021-04-12 17:55:07,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x6d type=client open_count=1 channel=[id: 0x616a2812, 0.0.0.0/0.0.0.0:55160] timeout=300.0>) 2021-04-12 17:55:09,012 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x6e type=client open_count=1 channel=[id: 0x632f2232, 0.0.0.0/0.0.0.0:57641] timeout=300.0>) 2021-04-12 17:55:11,023 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x6f type=client open_count=1 channel=[id: 0xe74c6647, 0.0.0.0/0.0.0.0:45357] timeout=300.0>) 2021-04-12 17:55:13,033 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x70 type=client open_count=1 channel=[id: 0x1c60e71f, 0.0.0.0/0.0.0.0:33404] timeout=300.0>) 2021-04-12 17:55:15,046 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x71 type=client open_count=1 channel=[id: 0x6e0dea63, 0.0.0.0/0.0.0.0:44194] timeout=300.0>) 2021-04-12 17:55:17,056 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x72 type=client open_count=1 channel=[id: 0x3d04d297, 0.0.0.0/0.0.0.0:57451] timeout=300.0>) 2021-04-12 17:55:19,069 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x73 type=client open_count=1 channel=[id: 0x545ee597, 0.0.0.0/0.0.0.0:37915] timeout=300.0>) 2021-04-12 17:55:21,079 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x74 type=client open_count=1 channel=[id: 0x790a8453, 0.0.0.0/0.0.0.0:46928] timeout=300.0>) 2021-04-12 17:55:23,092 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x75 type=client open_count=1 channel=[id: 0x805089df, 0.0.0.0/0.0.0.0:47423] timeout=300.0>) 2021-04-12 17:55:25,102 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x76 type=client open_count=1 channel=[id: 0xa1aacb14, 0.0.0.0/0.0.0.0:51603] timeout=300.0>) 2021-04-12 17:55:27,115 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x77 type=client open_count=1 channel=[id: 0x4406c552, 0.0.0.0/0.0.0.0:52989] timeout=300.0>) 2021-04-12 17:55:29,125 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x78 type=client open_count=1 channel=[id: 0xb31ec21f, 0.0.0.0/0.0.0.0:54591] timeout=300.0>) 2021-04-12 17:55:31,137 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x79 type=client open_count=1 channel=[id: 0xc079debb, 0.0.0.0/0.0.0.0:35511] timeout=300.0>) 2021-04-12 17:55:33,148 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x7a type=client open_count=1 channel=[id: 0xada6d1e8, 0.0.0.0/0.0.0.0:35123] timeout=300.0>) 2021-04-12 17:55:35,161 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x7b type=client open_count=1 channel=[id: 0x02dcc769, 0.0.0.0/0.0.0.0:51408] timeout=300.0>) 2021-04-12 17:55:37,173 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x7c type=client open_count=1 channel=[id: 0x6334880d, 0.0.0.0/0.0.0.0:53453] timeout=300.0>) 2021-04-12 17:55:39,184 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x7d type=client open_count=1 channel=[id: 0x8d363643, 0.0.0.0/0.0.0.0:54921] timeout=300.0>) 2021-04-12 17:55:41,196 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x7e type=client open_count=1 channel=[id: 0xb0c57a85, 0.0.0.0/0.0.0.0:44826] timeout=300.0>) 2021-04-12 17:55:43,207 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x7f type=client open_count=1 channel=[id: 0x2d07e01f, 0.0.0.0/0.0.0.0:51015] timeout=300.0>) 2021-04-12 17:55:45,219 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x80 type=client open_count=1 channel=[id: 0x40acaa19, 0.0.0.0/0.0.0.0:52447] timeout=300.0>) 2021-04-12 17:55:47,230 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x81 type=client open_count=1 channel=[id: 0x04c06153, 0.0.0.0/0.0.0.0:34963] timeout=300.0>) 2021-04-12 17:55:49,243 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x82 type=client open_count=1 channel=[id: 0x84eb1812, 0.0.0.0/0.0.0.0:53451] timeout=300.0>) 2021-04-12 17:55:51,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x83 type=client open_count=1 channel=[id: 0x6369ac8b, 0.0.0.0/0.0.0.0:54417] timeout=300.0>) 2021-04-12 17:55:53,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x84 type=client open_count=1 channel=[id: 0x9d8f8333, 0.0.0.0/0.0.0.0:56895] timeout=300.0>) 2021-04-12 17:55:55,279 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x85 type=client open_count=1 channel=[id: 0x6acf12dc, 0.0.0.0/0.0.0.0:38186] timeout=300.0>) 2021-04-12 17:55:57,292 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x86 type=client open_count=1 channel=[id: 0x12207fe9, 0.0.0.0/0.0.0.0:35367] timeout=300.0>) 2021-04-12 17:55:59,303 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x87 type=client open_count=1 channel=[id: 0xe4733919, 0.0.0.0/0.0.0.0:57652] timeout=300.0>) 2021-04-12 17:56:01,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x88 type=client open_count=1 channel=[id: 0xd5935fc0, 0.0.0.0/0.0.0.0:53605] timeout=300.0>) 2021-04-12 17:56:03,328 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x89 type=client open_count=1 channel=[id: 0xe25f8e65, 0.0.0.0/0.0.0.0:36994] timeout=300.0>) 2021-04-12 17:56:05,341 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x8a type=client open_count=1 channel=[id: 0xdd859c19, 0.0.0.0/0.0.0.0:59274] timeout=300.0>) 2021-04-12 17:56:07,354 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x8b type=client open_count=1 channel=[id: 0x975c1809, 0.0.0.0/0.0.0.0:55248] timeout=300.0>) 2021-04-12 17:56:09,365 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x8c type=client open_count=1 channel=[id: 0x7e95b81a, 0.0.0.0/0.0.0.0:44405] timeout=300.0>) 2021-04-12 17:56:11,378 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x8d type=client open_count=1 channel=[id: 0xfbb7a687, 0.0.0.0/0.0.0.0:35795] timeout=300.0>) 2021-04-12 17:56:13,391 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x8e type=client open_count=1 channel=[id: 0xcc13cde5, 0.0.0.0/0.0.0.0:47459] timeout=300.0>) 2021-04-12 17:56:15,404 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x8f type=client open_count=1 channel=[id: 0x75b5c7ef, 0.0.0.0/0.0.0.0:54167] timeout=300.0>) 2021-04-12 17:56:17,415 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x90 type=client open_count=1 channel=[id: 0xe8113b0b, 0.0.0.0/0.0.0.0:57571] timeout=300.0>) 2021-04-12 17:56:19,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x91 type=client open_count=1 channel=[id: 0x88390b70, 0.0.0.0/0.0.0.0:52838] timeout=300.0>) 2021-04-12 17:56:21,440 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x92 type=client open_count=1 channel=[id: 0xb55a2042, 0.0.0.0/0.0.0.0:52365] timeout=300.0>) 2021-04-12 17:56:23,451 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x93 type=client open_count=1 channel=[id: 0x04956a31, 0.0.0.0/0.0.0.0:33542] timeout=300.0>) 2021-04-12 17:56:25,463 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x94 type=client open_count=1 channel=[id: 0xde5aa856, 0.0.0.0/0.0.0.0:38803] timeout=300.0>) 2021-04-12 17:56:27,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x95 type=client open_count=1 channel=[id: 0xfedfe961, 0.0.0.0/0.0.0.0:45809] timeout=300.0>) 2021-04-12 17:56:29,487 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x96 type=client open_count=1 channel=[id: 0xfd9d4a5d, 0.0.0.0/0.0.0.0:60136] timeout=300.0>) 2021-04-12 17:56:31,499 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x97 type=client open_count=1 channel=[id: 0xbbcdc98c, 0.0.0.0/0.0.0.0:56132] timeout=300.0>) 2021-04-12 17:56:33,510 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x98 type=client open_count=1 channel=[id: 0xb5ae7b2e, 0.0.0.0/0.0.0.0:49932] timeout=300.0>) 2021-04-12 17:56:35,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x99 type=client open_count=1 channel=[id: 0x7d86d730, 0.0.0.0/0.0.0.0:39431] timeout=300.0>) 2021-04-12 17:56:37,535 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x9a type=client open_count=1 channel=[id: 0x32058213, 0.0.0.0/0.0.0.0:41295] timeout=300.0>) 2021-04-12 17:56:39,548 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x9b type=client open_count=1 channel=[id: 0xcef3b090, 0.0.0.0/0.0.0.0:33098] timeout=300.0>) 2021-04-12 17:56:41,559 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x9c type=client open_count=1 channel=[id: 0x585964bd, 0.0.0.0/0.0.0.0:43235] timeout=300.0>) 2021-04-12 17:56:43,573 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x9d type=client open_count=1 channel=[id: 0x1c4a14c7, 0.0.0.0/0.0.0.0:40421] timeout=300.0>) 2021-04-12 17:56:45,586 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x9e type=client open_count=1 channel=[id: 0x8cc41e78, 0.0.0.0/0.0.0.0:42221] timeout=300.0>) 2021-04-12 17:56:47,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x9f type=client open_count=1 channel=[id: 0xc926901f, 0.0.0.0/0.0.0.0:53165] timeout=300.0>) 2021-04-12 17:56:49,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa0 type=client open_count=1 channel=[id: 0x7285969f, 0.0.0.0/0.0.0.0:47755] timeout=300.0>) 2021-04-12 17:56:51,624 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa1 type=client open_count=1 channel=[id: 0xd9a419a3, 0.0.0.0/0.0.0.0:51578] timeout=300.0>) 2021-04-12 17:56:53,634 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa2 type=client open_count=1 channel=[id: 0x5a5b0a51, 0.0.0.0/0.0.0.0:40513] timeout=300.0>) 2021-04-12 17:56:55,645 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa3 type=client open_count=1 channel=[id: 0xd1d775f2, 0.0.0.0/0.0.0.0:34418] timeout=300.0>) 2021-04-12 17:56:57,657 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa4 type=client open_count=1 channel=[id: 0xdd76dd64, 0.0.0.0/0.0.0.0:47088] timeout=300.0>) 2021-04-12 17:56:59,668 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa5 type=client open_count=1 channel=[id: 0xdf1d7356, 0.0.0.0/0.0.0.0:43240] timeout=300.0>) 2021-04-12 17:57:01,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa6 type=client open_count=1 channel=[id: 0x74700699, 0.0.0.0/0.0.0.0:60250] timeout=300.0>) 2021-04-12 17:57:03,690 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa7 type=client open_count=1 channel=[id: 0x3e085314, 0.0.0.0/0.0.0.0:38913] timeout=300.0>) 2021-04-12 17:57:05,704 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa8 type=client open_count=1 channel=[id: 0x1b3f8b11, 0.0.0.0/0.0.0.0:47827] timeout=300.0>) 2021-04-12 17:57:07,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xa9 type=client open_count=1 channel=[id: 0x3fa9f405, 0.0.0.0/0.0.0.0:37428] timeout=300.0>) 2021-04-12 17:57:09,729 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xaa type=client open_count=1 channel=[id: 0x69d5bd00, 0.0.0.0/0.0.0.0:42413] timeout=300.0>) 2021-04-12 17:57:11,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xab type=client open_count=1 channel=[id: 0x46bd9265, 0.0.0.0/0.0.0.0:37066] timeout=300.0>) 2021-04-12 17:57:13,756 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xac type=client open_count=1 channel=[id: 0x0b38319f, 0.0.0.0/0.0.0.0:41340] timeout=300.0>) 2021-04-12 17:57:15,766 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xad type=client open_count=1 channel=[id: 0x9a0d30ea, 0.0.0.0/0.0.0.0:50328] timeout=300.0>) 2021-04-12 17:57:17,778 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xae type=client open_count=1 channel=[id: 0x76b4a267, 0.0.0.0/0.0.0.0:60803] timeout=300.0>) 2021-04-12 17:57:19,789 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xaf type=client open_count=1 channel=[id: 0x5a9bff18, 0.0.0.0/0.0.0.0:41009] timeout=300.0>) 2021-04-12 17:57:21,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb0 type=client open_count=1 channel=[id: 0xfec42d7b, 0.0.0.0/0.0.0.0:35717] timeout=300.0>) 2021-04-12 17:57:23,811 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb1 type=client open_count=1 channel=[id: 0x3ae050d3, 0.0.0.0/0.0.0.0:50640] timeout=300.0>) 2021-04-12 17:57:25,822 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb2 type=client open_count=1 channel=[id: 0x22fda965, 0.0.0.0/0.0.0.0:46557] timeout=300.0>) 2021-04-12 17:57:27,832 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb3 type=client open_count=1 channel=[id: 0x85ecfb35, 0.0.0.0/0.0.0.0:54188] timeout=300.0>) 2021-04-12 17:57:29,844 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb4 type=client open_count=1 channel=[id: 0xfd2d0005, 0.0.0.0/0.0.0.0:54480] timeout=300.0>) 2021-04-12 17:57:31,857 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb5 type=client open_count=1 channel=[id: 0x73132aa1, 0.0.0.0/0.0.0.0:52152] timeout=300.0>) 2021-04-12 17:57:33,869 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb6 type=client open_count=1 channel=[id: 0x3890c4c5, 0.0.0.0/0.0.0.0:49493] timeout=300.0>) 2021-04-12 17:57:35,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb7 type=client open_count=1 channel=[id: 0xd9830618, 0.0.0.0/0.0.0.0:47885] timeout=300.0>) 2021-04-12 17:57:37,891 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb8 type=client open_count=1 channel=[id: 0x549e7a11, 0.0.0.0/0.0.0.0:34294] timeout=300.0>) 2021-04-12 17:57:39,904 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xb9 type=client open_count=1 channel=[id: 0xd902b3ca, 0.0.0.0/0.0.0.0:38753] timeout=300.0>) 2021-04-12 17:57:41,914 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xba type=client open_count=1 channel=[id: 0xc5c13a06, 0.0.0.0/0.0.0.0:43586] timeout=300.0>) 2021-04-12 17:57:43,927 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xbb type=client open_count=1 channel=[id: 0x8e3892f0, 0.0.0.0/0.0.0.0:40352] timeout=300.0>) 2021-04-12 17:57:45,938 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xbc type=client open_count=1 channel=[id: 0x8d8db669, 0.0.0.0/0.0.0.0:40806] timeout=300.0>) 2021-04-12 17:57:47,950 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xbd type=client open_count=1 channel=[id: 0x137ebed8, 0.0.0.0/0.0.0.0:57214] timeout=300.0>) 2021-04-12 17:57:49,963 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xbe type=client open_count=1 channel=[id: 0x0c21c3da, 0.0.0.0/0.0.0.0:41192] timeout=300.0>) 2021-04-12 17:57:51,976 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xbf type=client open_count=1 channel=[id: 0xe0032f41, 0.0.0.0/0.0.0.0:46295] timeout=300.0>) 2021-04-12 17:57:53,987 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xc0 type=client open_count=1 channel=[id: 0x7514e448, 0.0.0.0/0.0.0.0:38088] timeout=300.0>) 2021-04-12 17:57:56,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xc1 type=client open_count=1 channel=[id: 0x9f5909e8, 0.0.0.0/0.0.0.0:48477] timeout=300.0>) 2021-04-12 17:57:58,013 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xc2 type=client open_count=1 channel=[id: 0x8ec284a7, 0.0.0.0/0.0.0.0:60051] timeout=300.0>) 2021-04-12 17:58:00,025 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xc3 type=client open_count=1 channel=[id: 0x40d4f962, 0.0.0.0/0.0.0.0:55717] timeout=300.0>) 2021-04-12 17:58:02,036 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xc4 type=client open_count=1 channel=[id: 0x3a0de1e0, 0.0.0.0/0.0.0.0:45146] timeout=300.0>) 2021-04-12 17:58:04,048 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0xc5 type=client open_count=1 channel=[id: 0xb9579d2c, 0.0.0.0/0.0.0.0:34199] timeout=300.0>) 2021-04-12 17:58:06,059 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xc6 type=client open_count=1 channel=[id: 0x7e425b99, 0.0.0.0/0.0.0.0:52946] timeout=300.0>) 2021-04-12 17:59:46,437 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xc7 type=client open_count=1 channel=[id: 0x21e27ec1, 0.0.0.0/0.0.0.0:51095] timeout=300.0>) 2021-04-12 17:59:48,447 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xc8 type=client open_count=1 channel=[id: 0xa714967a, 0.0.0.0/0.0.0.0:58306] timeout=300.0>) 2021-04-12 17:59:50,459 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xc9 type=client open_count=1 channel=[id: 0x7cd020f8, 0.0.0.0/0.0.0.0:37471] timeout=300.0>) 2021-04-12 17:59:52,470 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xca type=client open_count=1 channel=[id: 0x0e2e8f02, 0.0.0.0/0.0.0.0:55992] timeout=300.0>) 2021-04-12 17:59:54,480 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xcb type=client open_count=1 channel=[id: 0xbf1713ae, 0.0.0.0/0.0.0.0:46059] timeout=300.0>) 2021-04-12 17:59:56,490 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xcc type=client open_count=1 channel=[id: 0x71e031cb, 0.0.0.0/0.0.0.0:43788] timeout=300.0>) 2021-04-12 17:59:58,500 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xcd type=client open_count=1 channel=[id: 0x9804d73f, 0.0.0.0/0.0.0.0:51461] timeout=300.0>) 2021-04-12 18:00:00,513 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xce type=client open_count=1 channel=[id: 0x914a43e2, 0.0.0.0/0.0.0.0:50894] timeout=300.0>) 2021-04-12 18:00:02,525 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xcf type=client open_count=1 channel=[id: 0xd1fc489f, 0.0.0.0/0.0.0.0:54028] timeout=300.0>) 2021-04-12 18:00:04,536 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd0 type=client open_count=1 channel=[id: 0x2ea66cde, 0.0.0.0/0.0.0.0:43443] timeout=300.0>) 2021-04-12 18:00:06,548 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd1 type=client open_count=1 channel=[id: 0xe9e68b8c, 0.0.0.0/0.0.0.0:52203] timeout=300.0>) 2021-04-12 18:00:08,559 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd2 type=client open_count=1 channel=[id: 0xf93d4e34, 0.0.0.0/0.0.0.0:33321] timeout=300.0>) 2021-04-12 18:00:10,572 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd3 type=client open_count=1 channel=[id: 0x718be407, 0.0.0.0/0.0.0.0:38368] timeout=300.0>) 2021-04-12 18:00:12,585 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd4 type=client open_count=1 channel=[id: 0x77d63bca, 0.0.0.0/0.0.0.0:42799] timeout=300.0>) 2021-04-12 18:00:14,595 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd5 type=client open_count=1 channel=[id: 0xdc1c32ff, 0.0.0.0/0.0.0.0:57059] timeout=300.0>) 2021-04-12 18:00:16,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd6 type=client open_count=1 channel=[id: 0x951169c9, 0.0.0.0/0.0.0.0:39470] timeout=300.0>) 2021-04-12 18:00:18,619 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd7 type=client open_count=1 channel=[id: 0xd4760a4f, 0.0.0.0/0.0.0.0:47312] timeout=300.0>) 2021-04-12 18:00:20,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd8 type=client open_count=1 channel=[id: 0xd4d6a3b7, 0.0.0.0/0.0.0.0:34197] timeout=300.0>) 2021-04-12 18:00:22,644 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xd9 type=client open_count=1 channel=[id: 0x32368beb, 0.0.0.0/0.0.0.0:56570] timeout=300.0>) 2021-04-12 18:00:24,657 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xda type=client open_count=1 channel=[id: 0xba0fe0c4, 0.0.0.0/0.0.0.0:60188] timeout=300.0>) 2021-04-12 18:00:26,667 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xdb type=client open_count=1 channel=[id: 0xac8b8855, 0.0.0.0/0.0.0.0:34996] timeout=300.0>) 2021-04-12 18:00:28,677 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xdc type=client open_count=1 channel=[id: 0x4f884963, 0.0.0.0/0.0.0.0:35286] timeout=300.0>) 2021-04-12 18:00:30,690 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xdd type=client open_count=1 channel=[id: 0xdb649054, 0.0.0.0/0.0.0.0:51850] timeout=300.0>) 2021-04-12 18:00:32,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xde type=client open_count=1 channel=[id: 0xc33d821c, 0.0.0.0/0.0.0.0:38400] timeout=300.0>) 2021-04-12 18:00:34,710 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xdf type=client open_count=1 channel=[id: 0x3836ede8, 0.0.0.0/0.0.0.0:55431] timeout=300.0>) 2021-04-12 18:00:36,721 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xe0 type=client open_count=1 channel=[id: 0x49fa4fa0, 0.0.0.0/0.0.0.0:59085] timeout=300.0>) 2021-04-12 18:00:38,733 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xe1 type=client open_count=1 channel=[id: 0x1a53b878, 0.0.0.0/0.0.0.0:36185] timeout=300.0>) 2021-04-12 18:00:40,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xe2 type=client open_count=1 channel=[id: 0x1eeee81d, 0.0.0.0/0.0.0.0:45372] timeout=300.0>) 2021-04-12 18:00:42,756 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xe3 type=client open_count=1 channel=[id: 0x0f0e784d, 0.0.0.0/0.0.0.0:42593] timeout=300.0>) 2021-04-12 18:00:44,792 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xe4 type=client open_count=1 channel=[id: 0x096c85d3, 0.0.0.0/0.0.0.0:39617] timeout=300.0>) 2021-04-12 18:00:46,803 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.192:8091 during connect (<_realsocket at 0xe5 type=client open_count=1 channel=[id: 0x0ee60d43, 0.0.0.0/0.0.0.0:33359] timeout=300.0>) 2021-04-12 18:00:48,813 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.192:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xe6 type=client open_count=1 channel=[id: 0x051e82e4, 0.0.0.0/0.0.0.0:52926] timeout=300.0>) 2021-04-12 18:01:39,361 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xe7 type=client open_count=1 channel=[id: 0x377e6598, 0.0.0.0/0.0.0.0:34296] timeout=300.0>) 2021-04-12 18:01:41,371 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xe8 type=client open_count=1 channel=[id: 0xbc305310, 0.0.0.0/0.0.0.0:35831] timeout=300.0>) 2021-04-12 18:01:43,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xe9 type=client open_count=1 channel=[id: 0xf34f2f78, 0.0.0.0/0.0.0.0:39907] timeout=300.0>) 2021-04-12 18:01:45,395 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xea type=client open_count=1 channel=[id: 0xd6627f64, 0.0.0.0/0.0.0.0:58371] timeout=300.0>) 2021-04-12 18:01:47,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xeb type=client open_count=1 channel=[id: 0x0da70f59, 0.0.0.0/0.0.0.0:44677] timeout=300.0>) 2021-04-12 18:01:49,418 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xec type=client open_count=1 channel=[id: 0x90d32ff3, 0.0.0.0/0.0.0.0:57055] timeout=300.0>) 2021-04-12 18:01:51,430 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xed type=client open_count=1 channel=[id: 0xcd38b522, 0.0.0.0/0.0.0.0:42521] timeout=300.0>) 2021-04-12 18:01:53,440 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xee type=client open_count=1 channel=[id: 0xb7db1e50, 0.0.0.0/0.0.0.0:46881] timeout=300.0>) 2021-04-12 18:01:55,453 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xef type=client open_count=1 channel=[id: 0x8a23766e, 0.0.0.0/0.0.0.0:49114] timeout=300.0>) 2021-04-12 18:01:57,464 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf0 type=client open_count=1 channel=[id: 0x3f53e640, 0.0.0.0/0.0.0.0:48487] timeout=300.0>) 2021-04-12 18:01:59,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf1 type=client open_count=1 channel=[id: 0xb571705c, 0.0.0.0/0.0.0.0:54312] timeout=300.0>) 2021-04-12 18:02:01,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf2 type=client open_count=1 channel=[id: 0x02262db1, 0.0.0.0/0.0.0.0:54550] timeout=300.0>) 2021-04-12 18:02:03,497 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf3 type=client open_count=1 channel=[id: 0x837572a2, 0.0.0.0/0.0.0.0:46199] timeout=300.0>) 2021-04-12 18:02:05,509 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf4 type=client open_count=1 channel=[id: 0x857dd7eb, 0.0.0.0/0.0.0.0:45244] timeout=300.0>) 2021-04-12 18:02:07,520 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf5 type=client open_count=1 channel=[id: 0xc9b407e8, 0.0.0.0/0.0.0.0:36791] timeout=300.0>) 2021-04-12 18:02:09,533 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf6 type=client open_count=1 channel=[id: 0x4731b805, 0.0.0.0/0.0.0.0:54042] timeout=300.0>) 2021-04-12 18:02:11,543 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf7 type=client open_count=1 channel=[id: 0xafe42f53, 0.0.0.0/0.0.0.0:53479] timeout=300.0>) 2021-04-12 18:02:13,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf8 type=client open_count=1 channel=[id: 0x05cf7699, 0.0.0.0/0.0.0.0:37861] timeout=300.0>) 2021-04-12 18:02:15,566 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xf9 type=client open_count=1 channel=[id: 0x67499b5f, 0.0.0.0/0.0.0.0:53364] timeout=300.0>) 2021-04-12 18:02:17,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xfa type=client open_count=1 channel=[id: 0x1a939eba, 0.0.0.0/0.0.0.0:36397] timeout=300.0>) 2021-04-12 18:02:19,588 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xfb type=client open_count=1 channel=[id: 0x2d7389f1, 0.0.0.0/0.0.0.0:55827] timeout=300.0>) 2021-04-12 18:02:21,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xfc type=client open_count=1 channel=[id: 0x25a10626, 0.0.0.0/0.0.0.0:39531] timeout=300.0>) 2021-04-12 18:02:23,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xfd type=client open_count=1 channel=[id: 0xbc8889ac, 0.0.0.0/0.0.0.0:52275] timeout=300.0>) 2021-04-12 18:02:25,622 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xfe type=client open_count=1 channel=[id: 0x224a04b8, 0.0.0.0/0.0.0.0:39513] timeout=300.0>) 2021-04-12 18:02:27,634 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0xff type=client open_count=1 channel=[id: 0xd4f8c083, 0.0.0.0/0.0.0.0:57200] timeout=300.0>) 2021-04-12 18:02:29,645 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0x100 type=client open_count=1 channel=[id: 0x5f712f1f, 0.0.0.0/0.0.0.0:48274] timeout=300.0>) 2021-04-12 18:02:31,657 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0x101 type=client open_count=1 channel=[id: 0x77f5878e, 0.0.0.0/0.0.0.0:41059] timeout=300.0>) 2021-04-12 18:02:33,668 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0x102 type=client open_count=1 channel=[id: 0xa62e88d4, 0.0.0.0/0.0.0.0:57539] timeout=300.0>) 2021-04-12 18:02:35,678 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.193:8091 during connect (<_realsocket at 0x103 type=client open_count=1 channel=[id: 0x80a67021, 0.0.0.0/0.0.0.0:54074] timeout=300.0>) 2021-04-12 18:02:37,690 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.193:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.195:8091 during connect (<_realsocket at 0x104 type=client open_count=1 channel=[id: 0xab9ae938, 0.0.0.0/0.0.0.0:41856] timeout=300.0>) 2021-04-12 18:03:25,565 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.195:8091 during connect (<_realsocket at 0x105 type=client open_count=1 channel=[id: 0x5258e194, 0.0.0.0/0.0.0.0:55030] timeout=300.0>) 2021-04-12 18:03:27,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.195:8091 during connect (<_realsocket at 0x106 type=client open_count=1 channel=[id: 0x49b3b150, 0.0.0.0/0.0.0.0:36359] timeout=300.0>) 2021-04-12 18:03:29,588 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.195:8091 during connect (<_realsocket at 0x107 type=client open_count=1 channel=[id: 0xb99e8339, 0.0.0.0/0.0.0.0:56367] timeout=300.0>) 2021-04-12 18:03:31,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.195:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x108 type=client open_count=1 channel=[id: 0x10612dce, 0.0.0.0/0.0.0.0:54299] timeout=300.0>) 2021-04-12 18:04:00,467 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x109 type=client open_count=1 channel=[id: 0x26e01adb, 0.0.0.0/0.0.0.0:46209] timeout=300.0>) 2021-04-12 18:04:02,477 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x10a type=client open_count=1 channel=[id: 0x04d6491a, 0.0.0.0/0.0.0.0:49992] timeout=300.0>) 2021-04-12 18:04:04,489 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x10b type=client open_count=1 channel=[id: 0xc28c47ed, 0.0.0.0/0.0.0.0:56133] timeout=300.0>) 2021-04-12 18:04:06,500 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x10c type=client open_count=1 channel=[id: 0x46a6e898, 0.0.0.0/0.0.0.0:52002] timeout=300.0>) 2021-04-12 18:04:08,512 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x10d type=client open_count=1 channel=[id: 0x9d6852f8, 0.0.0.0/0.0.0.0:48291] timeout=300.0>) 2021-04-12 18:04:10,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x10e type=client open_count=1 channel=[id: 0x3d1cc0b5, 0.0.0.0/0.0.0.0:34561] timeout=300.0>) 2021-04-12 18:04:12,535 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x10f type=client open_count=1 channel=[id: 0xf5cf106b, 0.0.0.0/0.0.0.0:59328] timeout=300.0>) 2021-04-12 18:04:14,546 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x110 type=client open_count=1 channel=[id: 0xd2c4d199, 0.0.0.0/0.0.0.0:51115] timeout=300.0>) 2021-04-12 18:04:16,556 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x111 type=client open_count=1 channel=[id: 0xbf628b5f, 0.0.0.0/0.0.0.0:50653] timeout=300.0>) 2021-04-12 18:04:18,568 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x112 type=client open_count=1 channel=[id: 0x94fc2353, 0.0.0.0/0.0.0.0:56526] timeout=300.0>) 2021-04-12 18:04:20,579 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x113 type=client open_count=1 channel=[id: 0x369f3eeb, 0.0.0.0/0.0.0.0:47916] timeout=300.0>) 2021-04-12 18:04:22,591 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x114 type=client open_count=1 channel=[id: 0x8b0f9156, 0.0.0.0/0.0.0.0:53737] timeout=300.0>) 2021-04-12 18:04:24,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x115 type=client open_count=1 channel=[id: 0x365910b7, 0.0.0.0/0.0.0.0:43358] timeout=300.0>) 2021-04-12 18:04:26,615 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x116 type=client open_count=1 channel=[id: 0x2fc85923, 0.0.0.0/0.0.0.0:38300] timeout=300.0>) 2021-04-12 18:04:28,625 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.194:8091 during connect (<_realsocket at 0x117 type=client open_count=1 channel=[id: 0x2374e0cf, 0.0.0.0/0.0.0.0:58653] timeout=300.0>) 2021-04-12 18:04:30,635 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.194:8091/nodes/self error [Errno 111] Connection refused 2021-04-12 18:05:00,438 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:591] RAM quota was defined less than 100 MB: 2021-04-12 18:05:18,154 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.136.191, 172.23.136.192, 172.23.136.193, 172.23.136.195, 172.23.136.194 initialized 2021-04-12 18:05:18,168 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 18:05:18,188 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 18:05:49,993 | test | INFO | pool-10-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.136.191 | kv | 7.0.0-4902-enterprise | 24.7116666667 | Cluster node | | 172.23.136.192 | None | | | <--- IN --- | | 172.23.136.193 | None | | | <--- IN --- | | 172.23.136.195 | None | | | <--- IN --- | | 172.23.136.194 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-04-12 18:05:50,194 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-12 18:06:00,250 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-12 18:06:00,266 | test | INFO | pool-10-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2749998569 sec 2021-04-12 18:06:00,483 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-04-12 18:06:48,986 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-04-12 18:06:49,141 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 524288000 | 178622104 | 9575048 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 524288000 | 98991744 | 170 | | default | couchbase | 3 | none | 0 | 0 | 7864320000 | 641654696 | 93485 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-04-12 18:06:49,141 | test | INFO | MainThread | [collections_base:collection_setup:162] Creating required SDK clients for client_pool 2021-04-12 18:07:02,440 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-12 18:07:02,454 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-12 18:07:16,440 | test | INFO | pool-10-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................**X | 9023.298 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-04-12 18:07:18,188 | test | INFO | pool-10-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****....................................................................................................X | 2998.324 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-04-12 18:07:18,253 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.136.192 | kv | 80.178947017 | 6437847040 | 2726240256 | 4082102272 / 7511588864 | 111251 / 195927 | 7.0.0-4902-enterprise | | 172.23.136.194 | kv | 79.139028699 | 6437847040 | 3640410112 | 3112517632 / 7511588864 | 108553 / 261002 | 7.0.0-4902-enterprise | | 172.23.136.191 | kv | 85.0736334138 | 6437847040 | 3286011904 | 3720548352 / 7691014144 | 108946 / 254203 | 7.0.0-4902-enterprise | | 172.23.136.193 | kv | 79.036317272 | 6437847040 | 3647451136 | 3113652224 / 7511588864 | 110050 / 271217 | 7.0.0-4902-enterprise | | 172.23.136.195 | kv | 79.7104809858 | 6437847040 | 3811860480 | 2917732352 / 7511588864 | 109483 / 282815 | 7.0.0-4902-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-04-12 18:14:46,684 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-04-12 18:15:55,128 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 524288000 | 189613704 | 90290524 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 524288000 | 125100080 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 7864320000 | 1026829000 | 462362222 | +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ 2021-04-12 18:15:55,130 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 18:16:06,611 | test | INFO | MainThread | [collections_rebalance:setUp:57] Changing scrape interval to 2 2021-04-12 18:16:08,585 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 64} 2021-04-12 18:16:08,585 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load after hard_failover_recovery 2021-04-12 18:16:08,585 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : hard_failover_recovery 2021-04-12 18:16:08,586 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:647] failing over nodes [ip:172.23.136.195 port:8091 ssh_username:Administrator, ip:172.23.136.194 port:8091 ssh_username:Administrator] 2021-04-12 18:16:24,841 | test | INFO | pool-10-thread-5 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.67600011826 seconds to complete 2021-04-12 18:16:24,844 | test | INFO | pool-10-thread-5 | [common_lib:sleep:22] Sleep 8.67600011826 seconds. Reason: Wait after rebalance complete 2021-04-12 18:18:33,552 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:271] 1 nodes failed over as expected in 0.029000043869 seconds 2021-04-12 18:18:52,046 | test | INFO | pool-10-thread-17 | [rest_client:monitorRebalance:1446] Rebalance done. Taken 8.39100003242 seconds to complete 2021-04-12 18:18:52,049 | test | INFO | pool-10-thread-17 | [common_lib:sleep:22] Sleep 8.39100003242 seconds. Reason: Wait after rebalance complete 2021-04-12 18:21:00,471 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:271] 2 nodes failed over as expected in 0.0289998054504 seconds 2021-04-12 18:21:00,473 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:250] Starting a sync data load after failover 2021-04-12 18:21:00,476 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-12 18:21:00,483 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-12 18:21:09,108 | test | INFO | pool-10-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *........*.....................................................................*.......................X | 51660.319 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-04-12 18:21:09,720 | test | INFO | pool-10-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *............................................................*........................................X | 4296.032 | | 2649.183 | ............................................................X | 2649.183 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-04-12 18:21:11,105 | test | INFO | pool-10-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..................................................................................*..................X | 4510.576 | | 1124.877 | .......................*.........................................X | 2954.049 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-04-12 18:21:11,180 | test | WARNING | MainThread | [rest_client:get_nodes:1718] 172.23.136.194 - Node not part of cluster inactiveFailed 2021-04-12 18:21:11,180 | test | WARNING | MainThread | [rest_client:get_nodes:1718] 172.23.136.195 - Node not part of cluster inactiveFailed 2021-04-12 18:21:11,203 | test | WARNING | MainThread | [rest_client:get_nodes:1718] 172.23.136.194 - Node not part of cluster inactiveFailed 2021-04-12 18:21:11,203 | test | WARNING | MainThread | [rest_client:get_nodes:1718] 172.23.136.195 - Node not part of cluster inactiveFailed 2021-04-12 18:24:14,467 | test | INFO | pool-10-thread-13 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.136.192 | kv | 7.0.0-4902-enterprise | 11.0666666667 | Cluster node | | 172.23.136.194 | kv | 7.0.0-4902-enterprise | 0.783268060995 | Cluster node | | 172.23.136.191 | kv | 7.0.0-4902-enterprise | 37.7572273598 | Cluster node | | 172.23.136.193 | kv | 7.0.0-4902-enterprise | 10.6898850192 | Cluster node | | 172.23.136.195 | kv | 7.0.0-4902-enterprise | 0.391699308276 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-04-12 18:24:14,477 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-12 18:24:19,512 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-12 18:24:24,540 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-12 18:24:29,569 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-12 18:24:34,598 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-12 18:24:39,644 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-12 18:24:44,687 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 4.43 2021-04-12 18:24:49,753 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 9.07 2021-04-12 18:24:54,790 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 10.36 2021-04-12 18:24:59,829 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 11.55 2021-04-12 18:25:04,867 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 13.07 2021-04-12 18:25:09,901 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 14.52 2021-04-12 18:25:14,936 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 15.77 2021-04-12 18:25:19,970 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 17.56 2021-04-12 18:25:25,003 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 18.77 2021-04-12 18:25:30,040 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 21.78 2021-04-12 18:25:35,075 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 23.15 2021-04-12 18:25:40,115 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 24.13 2021-04-12 18:25:45,153 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 26.65 2021-04-12 18:25:50,186 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 27.87 2021-04-12 18:25:55,219 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 29.92 2021-04-12 18:26:00,257 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 30.71 2021-04-12 18:26:05,289 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 31.67 2021-04-12 18:26:10,322 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 33.06 2021-04-12 18:26:15,354 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-04-12 18:26:20,943 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-04-12 18:26:29,002 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 42.88 2021-04-12 18:26:34,694 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 65.47 2021-04-12 18:26:39,743 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-04-12 18:26:44,815 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.13 2021-04-12 18:26:49,865 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 74.31 2021-04-12 18:26:54,905 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 75.47 2021-04-12 18:26:59,944 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 76.17 2021-04-12 18:27:04,986 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 76.98 2021-04-12 18:27:10,026 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 78.74 2021-04-12 18:27:15,069 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 80.03 2021-04-12 18:27:20,108 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 80.99 2021-04-12 18:27:25,150 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 82.17 2021-04-12 18:27:30,191 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 84.22 2021-04-12 18:27:35,233 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 85.33 2021-04-12 18:27:40,285 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 86.56 2021-04-12 18:27:45,326 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 87.74 2021-04-12 18:27:50,378 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 89.25 2021-04-12 18:27:55,418 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 91.2 2021-04-12 18:28:00,460 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 92.32 2021-04-12 18:28:05,526 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 93.64 2021-04-12 18:28:10,566 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 94.67 2021-04-12 18:28:15,611 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 95.68 2021-04-12 18:28:20,658 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 96.23 2021-04-12 18:28:25,696 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 97.57 2021-04-12 18:28:30,737 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 98.94 2021-04-12 18:28:35,776 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: running, progress: 99.96 2021-04-12 18:33:40,805 | test | ERROR | pool-10-thread-13 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/pools/default/tasks error timed out 2021-04-12 18:33:40,806 | test | ERROR | pool-10-thread-13 | [rest_client:_rebalance_status_and_progress:1493] Unable to reach the host @ 172.23.136.191 2021-04-12 18:33:40,806 | test | INFO | pool-10-thread-13 | [task:check:355] Rebalance - status: None, progress: -100 2021-04-12 18:38:40,809 | test | ERROR | pool-10-thread-13 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/pools/default error timed out 2021-04-12 18:38:40,811 | test | ERROR | pool-10-thread-13 | [task:call:267] Unable to reach the host @ 172.23.136.191 FAIL 2021-04-12 18:38:40,812 | test | INFO | MainThread | [collections_rebalance:tearDown:78] Reverting prometheus settings back to default 2021-04-12 18:43:40,815 | infra | ERROR | MainThread | [Rest_Connection:_http_request:257] Socket error while connecting to http://172.23.136.191:8091/nodes/self. Error timed out ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 79, in tearDown StatsHelper(self.cluster.master).reset_stats_settings_from_diag_eval() File "lib/StatsLib/StatsOperations_Rest.py", line 17, in __init__ super(StatsHelper, self).__init__(server) File "./connections/Rest_Connection.py", line 122, in __init__ http_res, success = \ File "./connections/Rest_Connection.py", line 141, in init_http_request status, content, header = self._http_request( File "./connections/Rest_Connection.py", line 141, in init_http_request status, content, header = self._http_request( During the test, File "./connections/Rest_Connection.py", line 260, in _http_request raise ServerUnavailableException(ip=self.ip) ServerUnavailableException: Unable to reach the host @ 172.23.136.191 Remote Connections: 120, Disconnections: 120 SDK Connections: 22, Disconnections: 0 !!!!!! CRITICAL :: SDK disconnection mismatch !!!!! ====================================================================== FAIL: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 929, in test_data_load_collections_with_hard_failover_recovery self.load_collections_with_rebalance(rebalance_operation="hard_failover_recovery") File "pytests/bucket_collections/collections_rebalance.py", line 892, in load_collections_with_rebalance self.wait_for_rebalance_to_complete(rebalance) File "pytests/bucket_collections/collections_rebalance.py", line 792, in wait_for_rebalance_to_complete self.assertTrue(task.result, "Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 3212.125s FAILED (failures=1, errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 9 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_9 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_10 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_10', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 10, 'scrape_interval': '5'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 18:48:49,459 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #10 test_data_load_collections_with_hard_failover_recovery ========= Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16161 type=client open_count=1 channel=[id: 0x319b6c5a, 0.0.0.0/0.0.0.0:60427] timeout=300.0>) 2021-04-12 18:53:40,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16162 type=client open_count=1 channel=[id: 0x6bc39de4, 0.0.0.0/0.0.0.0:57561] timeout=300.0>) 2021-04-12 18:53:42,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16163 type=client open_count=1 channel=[id: 0x70abd0d9, 0.0.0.0/0.0.0.0:55205] timeout=300.0>) 2021-04-12 18:53:44,496 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16164 type=client open_count=1 channel=[id: 0xb5d4f362, 0.0.0.0/0.0.0.0:34839] timeout=300.0>) 2021-04-12 18:53:46,506 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16165 type=client open_count=1 channel=[id: 0xb29cfed2, 0.0.0.0/0.0.0.0:44887] timeout=300.0>) 2021-04-12 18:53:48,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16166 type=client open_count=1 channel=[id: 0x8a52a586, 0.0.0.0/0.0.0.0:47687] timeout=300.0>) 2021-04-12 18:53:50,528 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 260, in setUp self.cluster_util = ClusterUtils(self.cluster, self.task_manager) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 73, in __init__ self.rest = RestConnection(cluster.master) File "lib/membase/api/rest_client.py", line 279, in __init__ http_res, success = self.init_http_request(self.baseUrl + File "lib/membase/api/rest_client.py", line 388, in init_http_request status, content, header = self._http_request(api, 'GET', File "lib/membase/api/rest_client.py", line 388, in init_http_request status, content, header = self._http_request(api, 'GET', File "lib/membase/api/rest_client.py", line 755, in _http_request raise ServerUnavailableException(ip=self.ip) ServerUnavailableException: Unable to reach the host @ 172.23.136.191 ---------------------------------------------------------------------- Ran 1 test in 609.693s FAILED (errors=1) During the test, Remote Connections: 5, Disconnections: 5 SDK Connections: 0, Disconnections: 0 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 10 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_10 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_11 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_11', 'nodes_init': '5', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 11} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 18:54:33,963 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16167 type=client open_count=1 channel=[id: 0x5d9e6514, 0.0.0.0/0.0.0.0:57899] timeout=300.0>) 2021-04-12 18:54:33,967 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16168 type=client open_count=1 channel=[id: 0x54bcbe08, 0.0.0.0/0.0.0.0:60261] timeout=300.0>) 2021-04-12 18:54:35,977 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16169 type=client open_count=1 channel=[id: 0x7ed8a7ca, 0.0.0.0/0.0.0.0:44364] timeout=300.0>) 2021-04-12 18:54:37,989 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1616a type=client open_count=1 channel=[id: 0x4cd7ce5e, 0.0.0.0/0.0.0.0:36533] timeout=300.0>) 2021-04-12 18:54:40,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1616b type=client open_count=1 channel=[id: 0xf9959114, 0.0.0.0/0.0.0.0:49716] timeout=300.0>) 2021-04-12 18:54:42,010 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1616c type=client open_count=1 channel=[id: 0xa4f46472, 0.0.0.0/0.0.0.0:43466] timeout=300.0>) 2021-04-12 18:54:44,022 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1616d type=client open_count=1 channel=[id: 0xa09df6d3, 0.0.0.0/0.0.0.0:43179] timeout=300.0>) 2021-04-12 18:54:46,035 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1616e type=client open_count=1 channel=[id: 0xdbec9694, 0.0.0.0/0.0.0.0:40167] timeout=300.0>) 2021-04-12 18:54:48,045 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1616f type=client open_count=1 channel=[id: 0xca2fa9c4, 0.0.0.0/0.0.0.0:47485] timeout=300.0>) 2021-04-12 18:54:50,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16170 type=client open_count=1 channel=[id: 0xbfd9dab1, 0.0.0.0/0.0.0.0:50419] timeout=300.0>) 2021-04-12 18:54:52,065 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16171 type=client open_count=1 channel=[id: 0xca7ac1d0, 0.0.0.0/0.0.0.0:55771] timeout=300.0>) 2021-04-12 18:54:54,076 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16172 type=client open_count=1 channel=[id: 0x3ff49cc5, 0.0.0.0/0.0.0.0:38576] timeout=300.0>) 2021-04-12 18:54:56,088 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16173 type=client open_count=1 channel=[id: 0xbfb6dba1, 0.0.0.0/0.0.0.0:53080] timeout=300.0>) 2021-04-12 18:54:58,098 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16174 type=client open_count=1 channel=[id: 0x1d32488d, 0.0.0.0/0.0.0.0:44901] timeout=300.0>) 2021-04-12 18:55:00,109 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16175 type=client open_count=1 channel=[id: 0x62d61b78, 0.0.0.0/0.0.0.0:40918] timeout=300.0>) 2021-04-12 18:55:02,119 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16176 type=client open_count=1 channel=[id: 0xa14dba81, 0.0.0.0/0.0.0.0:59337] timeout=300.0>) 2021-04-12 18:55:04,130 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16177 type=client open_count=1 channel=[id: 0x59e5093d, 0.0.0.0/0.0.0.0:52993] timeout=300.0>) 2021-04-12 18:55:06,141 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16178 type=client open_count=1 channel=[id: 0xbe314c01, 0.0.0.0/0.0.0.0:45763] timeout=300.0>) 2021-04-12 18:55:08,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16179 type=client open_count=1 channel=[id: 0x628399d7, 0.0.0.0/0.0.0.0:40540] timeout=300.0>) 2021-04-12 18:55:10,161 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1617a type=client open_count=1 channel=[id: 0xc378eece, 0.0.0.0/0.0.0.0:54409] timeout=300.0>) 2021-04-12 18:55:12,171 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1617b type=client open_count=1 channel=[id: 0xb4d4f9bd, 0.0.0.0/0.0.0.0:37025] timeout=300.0>) 2021-04-12 18:55:14,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1617c type=client open_count=1 channel=[id: 0x58360bbb, 0.0.0.0/0.0.0.0:40352] timeout=300.0>) 2021-04-12 18:55:16,190 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1617d type=client open_count=1 channel=[id: 0x2f08eccd, 0.0.0.0/0.0.0.0:44731] timeout=300.0>) 2021-04-12 18:55:18,200 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1617e type=client open_count=1 channel=[id: 0xb38618a3, 0.0.0.0/0.0.0.0:39445] timeout=300.0>) 2021-04-12 18:55:20,210 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1617f type=client open_count=1 channel=[id: 0x387ad75c, 0.0.0.0/0.0.0.0:48321] timeout=300.0>) 2021-04-12 18:55:22,220 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16180 type=client open_count=1 channel=[id: 0x7228199d, 0.0.0.0/0.0.0.0:53902] timeout=300.0>) 2021-04-12 18:55:24,230 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16181 type=client open_count=1 channel=[id: 0x368fc48d, 0.0.0.0/0.0.0.0:36191] timeout=300.0>) 2021-04-12 18:55:26,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16182 type=client open_count=1 channel=[id: 0xa14c816e, 0.0.0.0/0.0.0.0:39002] timeout=300.0>) 2021-04-12 18:55:28,250 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16183 type=client open_count=1 channel=[id: 0xd63dfde6, 0.0.0.0/0.0.0.0:52610] timeout=300.0>) 2021-04-12 18:55:30,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16184 type=client open_count=1 channel=[id: 0x1a9e253f, 0.0.0.0/0.0.0.0:46355] timeout=300.0>) 2021-04-12 18:55:32,270 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16185 type=client open_count=1 channel=[id: 0x18be0820, 0.0.0.0/0.0.0.0:57541] timeout=300.0>) 2021-04-12 18:55:34,280 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16186 type=client open_count=1 channel=[id: 0x823a378f, 0.0.0.0/0.0.0.0:52505] timeout=300.0>) 2021-04-12 18:55:36,290 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16187 type=client open_count=1 channel=[id: 0xd98e193d, 0.0.0.0/0.0.0.0:38357] timeout=300.0>) 2021-04-12 18:55:38,302 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16188 type=client open_count=1 channel=[id: 0xb8387e6a, 0.0.0.0/0.0.0.0:55595] timeout=300.0>) 2021-04-12 18:55:40,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16189 type=client open_count=1 channel=[id: 0x94ee5be0, 0.0.0.0/0.0.0.0:47946] timeout=300.0>) 2021-04-12 18:55:42,322 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1618a type=client open_count=1 channel=[id: 0x6bc438da, 0.0.0.0/0.0.0.0:57741] timeout=300.0>) 2021-04-12 18:55:44,335 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1618b type=client open_count=1 channel=[id: 0x518d93d8, 0.0.0.0/0.0.0.0:60484] timeout=300.0>) 2021-04-12 18:55:46,345 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1618c type=client open_count=1 channel=[id: 0x78d9679e, 0.0.0.0/0.0.0.0:55506] timeout=300.0>) 2021-04-12 18:55:48,358 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1618d type=client open_count=1 channel=[id: 0x4b82f7fd, 0.0.0.0/0.0.0.0:40278] timeout=300.0>) 2021-04-12 18:55:50,368 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1618e type=client open_count=1 channel=[id: 0x8ac16d19, 0.0.0.0/0.0.0.0:46319] timeout=300.0>) 2021-04-12 18:55:52,378 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1618f type=client open_count=1 channel=[id: 0x032b553f, 0.0.0.0/0.0.0.0:37066] timeout=300.0>) 2021-04-12 18:55:54,388 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16190 type=client open_count=1 channel=[id: 0x0fa5388d, 0.0.0.0/0.0.0.0:45498] timeout=300.0>) 2021-04-12 18:55:56,398 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16191 type=client open_count=1 channel=[id: 0x9d4ccefb, 0.0.0.0/0.0.0.0:51567] timeout=300.0>) 2021-04-12 18:55:58,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16192 type=client open_count=1 channel=[id: 0x3c3a7088, 0.0.0.0/0.0.0.0:36619] timeout=300.0>) 2021-04-12 18:56:00,418 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16193 type=client open_count=1 channel=[id: 0xa445d090, 0.0.0.0/0.0.0.0:38084] timeout=300.0>) 2021-04-12 18:56:02,428 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16194 type=client open_count=1 channel=[id: 0xc4f4509e, 0.0.0.0/0.0.0.0:43588] timeout=300.0>) 2021-04-12 18:56:04,438 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16195 type=client open_count=1 channel=[id: 0x99410432, 0.0.0.0/0.0.0.0:48151] timeout=300.0>) 2021-04-12 18:56:06,448 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16196 type=client open_count=1 channel=[id: 0x13782e57, 0.0.0.0/0.0.0.0:44980] timeout=300.0>) 2021-04-12 18:56:08,459 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16197 type=client open_count=1 channel=[id: 0xd9b162fd, 0.0.0.0/0.0.0.0:38197] timeout=300.0>) 2021-04-12 18:56:10,470 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16198 type=client open_count=1 channel=[id: 0x56a0d96b, 0.0.0.0/0.0.0.0:46438] timeout=300.0>) 2021-04-12 18:56:12,482 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x16199 type=client open_count=1 channel=[id: 0xf5f6b24d, 0.0.0.0/0.0.0.0:57071] timeout=300.0>) 2021-04-12 18:56:14,492 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1619a type=client open_count=1 channel=[id: 0x885b4b2c, 0.0.0.0/0.0.0.0:52742] timeout=300.0>) 2021-04-12 18:56:16,502 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1619b type=client open_count=1 channel=[id: 0x133c024c, 0.0.0.0/0.0.0.0:35953] timeout=300.0>) 2021-04-12 18:56:18,513 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1619c type=client open_count=1 channel=[id: 0xca5a5943, 0.0.0.0/0.0.0.0:59106] timeout=300.0>) 2021-04-12 18:56:20,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1619d type=client open_count=1 channel=[id: 0x89a701cc, 0.0.0.0/0.0.0.0:52829] timeout=300.0>) 2021-04-12 18:56:22,533 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1619e type=client open_count=1 channel=[id: 0xa0465f9e, 0.0.0.0/0.0.0.0:58622] timeout=300.0>) 2021-04-12 18:56:24,546 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x1619f type=client open_count=1 channel=[id: 0xea083b56, 0.0.0.0/0.0.0.0:49977] timeout=300.0>) 2021-04-12 18:56:26,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a0 type=client open_count=1 channel=[id: 0x1ee915a9, 0.0.0.0/0.0.0.0:35904] timeout=300.0>) 2021-04-12 18:56:28,566 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a1 type=client open_count=1 channel=[id: 0x90a98f82, 0.0.0.0/0.0.0.0:56299] timeout=300.0>) 2021-04-12 18:56:30,576 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a2 type=client open_count=1 channel=[id: 0x232502bb, 0.0.0.0/0.0.0.0:53809] timeout=300.0>) 2021-04-12 18:56:32,588 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a3 type=client open_count=1 channel=[id: 0x16d98088, 0.0.0.0/0.0.0.0:38377] timeout=300.0>) 2021-04-12 18:56:34,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a4 type=client open_count=1 channel=[id: 0xbc8b45d2, 0.0.0.0/0.0.0.0:52403] timeout=300.0>) 2021-04-12 18:56:36,609 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a5 type=client open_count=1 channel=[id: 0x57a72317, 0.0.0.0/0.0.0.0:53961] timeout=300.0>) 2021-04-12 18:56:38,619 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a6 type=client open_count=1 channel=[id: 0xd559a483, 0.0.0.0/0.0.0.0:40557] timeout=300.0>) 2021-04-12 18:56:40,630 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a7 type=client open_count=1 channel=[id: 0x6f2bc865, 0.0.0.0/0.0.0.0:34618] timeout=300.0>) 2021-04-12 18:56:42,638 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a8 type=client open_count=1 channel=[id: 0x9fd6152b, 0.0.0.0/0.0.0.0:56371] timeout=300.0>) 2021-04-12 18:56:44,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161a9 type=client open_count=1 channel=[id: 0x8bfcb586, 0.0.0.0/0.0.0.0:58171] timeout=300.0>) 2021-04-12 18:56:46,658 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161aa type=client open_count=1 channel=[id: 0xbe9c386a, 0.0.0.0/0.0.0.0:38914] timeout=300.0>) 2021-04-12 18:56:48,667 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161ab type=client open_count=1 channel=[id: 0x99307c01, 0.0.0.0/0.0.0.0:47250] timeout=300.0>) 2021-04-12 18:56:50,677 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161ac type=client open_count=1 channel=[id: 0x7dc475d2, 0.0.0.0/0.0.0.0:48281] timeout=300.0>) 2021-04-12 18:56:52,688 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161ad type=client open_count=1 channel=[id: 0x8a08c7a0, 0.0.0.0/0.0.0.0:55478] timeout=300.0>) 2021-04-12 18:56:54,698 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161ae type=client open_count=1 channel=[id: 0xbd13afda, 0.0.0.0/0.0.0.0:43292] timeout=300.0>) 2021-04-12 18:56:56,709 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.191:8091 during connect (<_realsocket at 0x161af type=client open_count=1 channel=[id: 0x500ab588, 0.0.0.0/0.0.0.0:46785] timeout=300.0>) 2021-04-12 18:56:58,719 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.191:8091/nodes/self error [Errno 111] Connection refused 2021-04-12 18:57:05,092 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 18:57:42,359 | infra | ERROR | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.136.191:8091/pools/default/buckets/bucket1 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-04-12 18:57:50,635 | infra | ERROR | MainThread | [remote_util:log_command_output:3065] wc: '/cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/data': Is a directory 2021-04-12 18:57:50,637 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1510] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-04-12 18:58:25,723 | infra | ERROR | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.136.191:8091/pools/default/buckets/bucket2 body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-04-12 18:58:40,095 | infra | ERROR | MainThread | [remote_util:log_command_output:3065] wc: '/cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/data': Is a directory 2021-04-12 18:58:40,096 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1510] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-04-12 19:00:08,178 | infra | ERROR | MainThread | [Rest_Connection:_http_request:253] DELETE http://172.23.136.191:8091/pools/default/buckets/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown {"_":"Bucket deletion not yet complete, but will continue.\r\n"} auth: Administrator:password 2021-04-12 19:00:23,447 | infra | ERROR | MainThread | [remote_util:log_command_output:3065] wc: '/cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/data': Is a directory 2021-04-12 19:00:23,506 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1510] Unable to get timings for bucket: global name 'StatsCommon' is not defined 2021-04-12 19:02:29,206 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'002e2153af12b06d826d26c75e3897d8', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 19:02:29,512 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279348020L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:02:28.020Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.136.191\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default","bucket1"]}}}]}.\nRebalance Operation Id = f8b927f80602176366b99bdd3ce3c3ec'} 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279348019L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:02:28.019Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.136.191\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default",\n "bucket1"]}}}]\n'} 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279227537L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:00:27.537Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193',\n 'ns_1@172.23.136.195'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f8b927f80602176366b99bdd3ce3c3ec"} 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618279227504L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T19:00:27.504Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_couchdb_api', u'type': u'critical', u'node': u'ns_1@172.23.136.193', u'tstamp': 1618279185156L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:45.156Z', u'text': u'Unable to delete bucket database directory default\n{error,eacces}'} 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618279178482L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:38.482Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.136.191\' for deletion'} 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.193', u'tstamp': 1618279178446L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:38.446Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.136.193\' for deletion'} 2021-04-12 19:02:29,513 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.194', u'tstamp': 1618279178335L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:38.335Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.136.194\' for deletion'} 2021-04-12 19:02:29,515 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.195', u'tstamp': 1618279178282L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:38.282Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.136.195\' for deletion'} 2021-04-12 19:02:29,515 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279178185L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:38.185Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.136.192\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'002e2153af12b06d826d26c75e3897d8', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'002e2153af12b06d826d26c75e3897d8', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 518.973s FAILED (errors=1) During the test, Remote Connections: 9, Disconnections: 9 SDK Connections: 0, Disconnections: 0 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 11 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_11 Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_12 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'after', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_12', 'nodes_init': '5', 'rebalance_moves_per_node': '64', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 12, 'scrape_interval': '2'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-12 19:02:35,552 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 19:02:36,871 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-04-12 19:04:39,855 | test | ERROR | MainThread | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'703d28536e32235b18858381f7f285cd', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed 2021-04-12 19:04:39,885 | test | INFO | MainThread | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.191: 2021-04-12 19:04:39,885 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279477134L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:04:37.134Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.136.191\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default"]}}}]}.\nRebalance Operation Id = 853eb96dc8f41ca1cf0b895ebcb1bcdb'} 2021-04-12 19:04:39,885 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279477133L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:04:37.133Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.136.191\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default"]}}}]\n'} 2021-04-12 19:04:39,885 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618279357267L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T19:02:37.267Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 19:04:39,887 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279357102L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:02:37.102Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193',\n 'ns_1@172.23.136.195'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 853eb96dc8f41ca1cf0b895ebcb1bcdb"} 2021-04-12 19:04:39,887 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279348020L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:02:28.020Z', u'text': u'Rebalance exited with reason {buckets_shutdown_wait_failed,\n [{\'ns_1@172.23.136.191\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default","bucket1"]}}}]}.\nRebalance Operation Id = f8b927f80602176366b99bdd3ce3c3ec'} 2021-04-12 19:04:39,887 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'critical', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279348019L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:02:28.019Z', u'text': u'Failed to wait deletion of some buckets on some nodes: [{\'ns_1@172.23.136.191\',\n {\'EXIT\',\n {old_buckets_shutdown_wait_failed,\n ["default",\n "bucket1"]}}}]\n'} 2021-04-12 19:04:39,887 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.192', u'tstamp': 1618279227537L, u'shortText': u'message', u'serverTime': u'2021-04-12T19:00:27.537Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.191'], EjectNodes = ['ns_1@172.23.136.192',\n 'ns_1@172.23.136.194',\n 'ns_1@172.23.136.193',\n 'ns_1@172.23.136.195'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f8b927f80602176366b99bdd3ce3c3ec"} 2021-04-12 19:04:39,887 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 102, u'module': u'menelaus_web', u'type': u'warning', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618279227504L, u'shortText': u'client-side error report', u'serverTime': u'2021-04-12T19:00:27.504Z', u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@172.23.136.191\':\nUser-Agent:Python-httplib2/$Rev: 259 $\nStarting rebalance from test, ejected nodes [u\'ns_1@172.23.136.192\', u\'ns_1@172.23.136.194\', u\'ns_1@172.23.136.193\', u\'ns_1@172.23.136.195\']'} 2021-04-12 19:04:39,888 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_couchdb_api', u'type': u'critical', u'node': u'ns_1@172.23.136.193', u'tstamp': 1618279185156L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:45.156Z', u'text': u'Unable to delete bucket database directory default\n{error,eacces}'} 2021-04-12 19:04:39,888 | test | ERROR | MainThread | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.191', u'tstamp': 1618279178482L, u'shortText': u'message', u'serverTime': u'2021-04-12T18:59:38.482Z', u'text': u'Shutting down bucket "default" on \'ns_1@172.23.136.191\' for deletion'} Traceback (most recent call last): File "pytests/basetestcase.py", line 296, in setUp cluster_util.cluster_cleanup(bucket_util) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 131, in cluster_cleanup self.cleanup_cluster(self.cluster.servers, master=self.cluster.master) File "couchbase_utils/cluster_utils/cluster_ready_functions.py", line 171, in cleanup_cluster ejectedNodes=[node.id for node in nodes File "lib/membase/api/rest_client.py", line 130, in remove_nodes return self.rest.monitorRebalance() File "lib/membase/api/rest_client.py", line 1416, in monitorRebalance progress = self._rebalance_progress() File "lib/membase/api/rest_client.py", line 1539, in _rebalance_progress return self._rebalance_status_and_progress()[1] File "lib/membase/api/rest_client.py", line 1512, in _rebalance_status_and_progress raise RebalanceFailedException(msg) RebalanceFailedException: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'703d28536e32235b18858381f7f285cd', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed During the test, Remote Connections: 6, Disconnections: 6 SDK Connections: 0, Disconnections: 0 ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 21, in setUp super(CollectionsRebalance, self).setUp() File "pytests/bucket_collections/collections_base.py", line 19, in setUp super(CollectionBase, self).setUp() File "pytests/basetestcase.py", line 372, in setUp self.fail(e) AssertionError: Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'703d28536e32235b18858381f7f285cd', u'statusIsStale': False, u'status': u'notRunning'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 130.357s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 12 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-12_17-45-13/test_12 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,GROUP=P0_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_failover_and_recovery' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY,step_count=1,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability= MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability= MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability= MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability= MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability= MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability= MAJORITY_AND_PERSIST_TO_ACTIVE,step_count=1,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,step_count=1,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,step_count=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,step_count=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,step_count=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,change_ram_quota_cluster=True,nodes_init=5,vbuckets=64,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_failover' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,change_ram_quota_cluster=True,nodes_init=5,vbuckets=64,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_failover' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,change_ram_quota_cluster=True,nodes_init=5,vbuckets=128,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_failover' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,change_ram_quota_cluster=True,nodes_init=5,vbuckets=128,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_failover' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,change_ram_quota_cluster=True,nodes_init=5,vbuckets=256,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_failover' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,change_ram_quota_cluster=True,nodes_init=5,vbuckets=512,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_failover' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_forced_hard_failover_rebalance_out,nodes_init=4,nodes_failover=2,forced_hard_failover=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_spec=initial_load,data_load_stage=during,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_forced_hard_failover_rebalance_out,nodes_init=5,nodes_failover=3,forced_hard_failover=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_spec=initial_load,data_load_stage=during,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 1h 19m 34s 2 actionable tasks: 1 executed, 1 up-to-date Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Could not determine description. [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE