Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-sc2402-32g-12c (volume_test_collections) in workspace /data/workspace/temp_rebalance_even_win_MB-45061 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_rebalance_even_win_MB-45061 # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision bda8d14a3459f56481eca83b0bcfd4e451217472 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f bda8d14a3459f56481eca83b0bcfd4e451217472 > /usr/bin/git rev-list ee12872bd8a6031b563ebf650b0cf4063dd4ee17 # timeout=10 [temp_rebalance_even_win_MB-45061] $ /bin/sh -xe /tmp/jenkins8955112002428776680.sh + echo '[global] username:Administrator password:Membase123 port:8091 index_port:9102 n1ql_port:8093 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 [_1] ip:172.23.136.128 services:kv [_2] ip:172.23.136.129 services:kv [_3] ip:172.23.136.252 services:kv [_4] ip:172.23.136.253 services:kv [_5] ip:172.23.136.254 services:kv' [temp_rebalance_even_win_MB-45061] $ /bin/sh -xe /tmp/jenkins859163363935280379.sh + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From https://github.com/couchbaselabs/TAF * branch master -> FETCH_HEAD Already up-to-date. + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + guides/gradlew check -s Starting a Gradle Daemon, 54 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 > Task :compileJava Note: /data/workspace/temp_rebalance_even_win_MB-45061/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :processResources > Task :classes > Task :compileTestJava NO-SOURCE > Task :processTestResources NO-SOURCE > Task :testClasses UP-TO-DATE > Task :test NO-SOURCE > Task :check UP-TO-DATE Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 5s 2 actionable tasks: 2 executed + guides/gradlew -v ------------------------------------------------------------ Gradle 6.2.2 ------------------------------------------------------------ Build time: 2020-03-04 08:49:31 UTC Revision: 7d0bf6dcb46c143bcc3b7a0fa40a8e5ca28e5856 Kotlin: 1.3.61 Groovy: 2.5.8 Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 JVM: 1.8.0_242 (Oracle Corporation 25.242-b08) OS: Linux 3.10.0-1062.18.1.el7.x86_64 amd64 + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_rebalance_even_win_MB-45061/build/classes/java/main:/data/workspace/temp_rebalance_even_win_MB-45061/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/win10-bucket-ops-11111222223334444.ini -p rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -c conf/collections/collections_failover.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_failover.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'GROUP': 'P0_failover_and_recovery', 'change_ephemeral_purge_age_and_interval': 'False', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'num_nodes': 5, 'rerun': 'False', 'spec': 'collections_failover'} Only cases in GROUPs 'P0_failover_and_recovery' will be executed Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_failover_and_rebalance_out' skipped, GROUP not satisfied Logs will be stored at /data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-16_23-05-18/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/win10-bucket-ops-11111222223334444.ini rerun=False,get-cbcollect-info=True,crash_warning=True,GROUP=P0_failover_and_recovery,change_ephemeral_purge_age_and_interval=False -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_failover_and_recovery' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'win10-bucket-ops-11111222223334444', 'nodes_failover': '2', 'ini': '/tmp/win10-bucket-ops-11111222223334444.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover.conf', 'crash_warning': 'True', 'change_ephemeral_purge_age_and_interval': 'False', 'skip_validations': 'False', 'spec': 'collections_failover', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_rebalance_even_win_MB-45061/logs/testrunner-21-Apr-16_23-05-18/test_1', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'P0_failover_and_recovery', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 1, 'scrape_interval': '5'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-04-16 23:05:41,431 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-16 23:05:42,242 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-16 23:05:42,793 | test | ERROR | MainThread | [rest_client:_http_request:748] GET http://172.23.136.128:8091/pools/default/ body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2021-04-16 23:05:43,043 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.136.128:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0x9bbdc483, 0.0.0.0/0.0.0.0:54135] timeout=300.0>) 2021-04-16 23:06:05,016 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.128:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.128:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x8589eb90, 0.0.0.0/0.0.0.0:58585] timeout=300.0>) 2021-04-16 23:06:07,039 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.128:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.128:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x17537598, 0.0.0.0/0.0.0.0:59785] timeout=300.0>) 2021-04-16 23:06:09,059 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.128:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.128:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0x729e8cc3, 0.0.0.0/0.0.0.0:51948] timeout=300.0>) 2021-04-16 23:06:11,076 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.128:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.129:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xfcda3d4c, 0.0.0.0/0.0.0.0:47927] timeout=300.0>) 2021-04-16 23:06:34,907 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.129:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0xbc9771cc, 0.0.0.0/0.0.0.0:45015] timeout=300.0>) 2021-04-16 23:06:36,924 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.129:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x5e1c54da, 0.0.0.0/0.0.0.0:52861] timeout=300.0>) 2021-04-16 23:06:38,940 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.129:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x2ed1ba2a, 0.0.0.0/0.0.0.0:43774] timeout=300.0>) 2021-04-16 23:06:40,957 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.252:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0xdb19696b, 0.0.0.0/0.0.0.0:54595] timeout=300.0>) 2021-04-16 23:07:04,971 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.252:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.252:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0xfac7e2d7, 0.0.0.0/0.0.0.0:41278] timeout=300.0>) 2021-04-16 23:07:06,986 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.252:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.252:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0xa24b4f96, 0.0.0.0/0.0.0.0:39158] timeout=300.0>) 2021-04-16 23:07:09,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.252:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.252:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x8571d638, 0.0.0.0/0.0.0.0:48516] timeout=300.0>) 2021-04-16 23:07:11,016 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.252:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.252:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x9fec9dd8, 0.0.0.0/0.0.0.0:35843] timeout=300.0>) 2021-04-16 23:07:13,029 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.252:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.253:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x9154d24a, 0.0.0.0/0.0.0.0:55220] timeout=300.0>) 2021-04-16 23:07:37,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.253:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.253:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0xd7eaae39, 0.0.0.0/0.0.0.0:54186] timeout=300.0>) 2021-04-16 23:07:39,585 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.253:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.253:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x6c9b17ed, 0.0.0.0/0.0.0.0:48606] timeout=300.0>) 2021-04-16 23:07:41,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.253:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.253:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0xe06dce2c, 0.0.0.0/0.0.0.0:51311] timeout=300.0>) 2021-04-16 23:07:43,614 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.253:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.254:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xd4b4fb5b, 0.0.0.0/0.0.0.0:56922] timeout=300.0>) 2021-04-16 23:08:08,033 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.254:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.254:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x62796ddc, 0.0.0.0/0.0.0.0:32883] timeout=300.0>) 2021-04-16 23:08:10,048 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.254:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.254:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0xc915f235, 0.0.0.0/0.0.0.0:55318] timeout=300.0>) 2021-04-16 23:08:12,061 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.254:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.254:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0x98d4d9a9, 0.0.0.0/0.0.0.0:49080] timeout=300.0>) 2021-04-16 23:08:14,075 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.254:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.254:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x7a4a3e42, 0.0.0.0/0.0.0.0:37625] timeout=300.0>) 2021-04-16 23:08:16,091 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.254:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.136.254:8091 during connect (<_realsocket at 0x23 type=client open_count=1 channel=[id: 0xd7f8eb66, 0.0.0.0/0.0.0.0:55836] timeout=300.0>) 2021-04-16 23:08:18,105 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.254:8091/nodes/self error [Errno 111] Connection refused 2021-04-16 23:08:35,614 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:591] RAM quota was defined less than 100 MB: 2021-04-16 23:08:49,721 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.136.128, 172.23.136.129, 172.23.136.252, 172.23.136.253, 172.23.136.254 initialized 2021-04-16 23:08:49,736 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-16 23:08:49,757 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-16 23:09:42,724 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.136.128 | kv | 7.0.0-4902-enterprise | 3.51345044835 | Cluster node | | 172.23.136.129 | None | | | <--- IN --- | | 172.23.136.252 | None | | | <--- IN --- | | 172.23.136.253 | None | | | <--- IN --- | | 172.23.136.254 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-04-16 23:09:42,740 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-16 23:09:52,788 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-16 23:09:52,803 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.0789999962 sec 2021-04-16 23:09:53,020 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-04-16 23:10:20,917 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-04-16 23:10:21,094 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 524288000 | 205922520 | 3831415 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 524288000 | 114648592 | 170 | | default | couchbase | 3 | none | 0 | 0 | 7864320000 | 981659512 | 990918 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-04-16 23:10:21,095 | test | INFO | MainThread | [collections_base:collection_setup:162] Creating required SDK clients for client_pool 2021-04-16 23:10:34,259 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-16 23:10:34,276 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-16 23:10:46,650 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.......................................................................*.............................***X | 367.884 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-04-16 23:10:48,012 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.****...................................................................................................X | 2391.894 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-04-16 23:10:48,084 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-------------+------------+--------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-------------+------------+--------------------------+-------------------+-----------------------+ | 172.23.136.253 | kv | 57.5398974957 | 11996348416 | 9373335552 | 3023761408 / 13808287744 | 661 / 1698 | 7.0.0-4902-enterprise | | 172.23.136.252 | kv | 59.9722104352 | 11996348416 | 9197969408 | 3133181952 / 13808287744 | 623 / 1775 | 7.0.0-4902-enterprise | | 172.23.136.128 | kv | 65.3531907813 | 11996348416 | 9118576640 | 3151572992 / 13808287744 | 527 / 1721 | 7.0.0-4902-enterprise | | 172.23.136.254 | kv | 58.7247091764 | 11996348416 | 9402789888 | 2963931136 / 13808287744 | 668 / 1669 | 7.0.0-4902-enterprise | | 172.23.136.129 | kv | 62.2079672524 | 11996348416 | 9131474944 | 3194478592 / 13808287744 | 974 / 3045 | 7.0.0-4902-enterprise | +----------------+----------+-----------------+-------------+------------+--------------------------+-------------------+-----------------------+ 2021-04-16 23:13:35,032 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4405] Validating collection stats and item counts 2021-04-16 23:14:33,415 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 524288000 | 205083096 | 90024486 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 524288000 | 136786224 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 7864320000 | 1042866136 | 454407499 | +---------+-----------+----------+------------+-----+--------+------------+------------+-----------+ 2021-04-16 23:14:33,418 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_graceful_failover_recovery ========= 2021-04-16 23:14:41,697 | test | INFO | MainThread | [collections_rebalance:setUp:57] Changing scrape interval to 5 2021-04-16 23:14:43,105 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:119] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-04-16 23:14:43,105 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:820] Doing collection data load before graceful_failover_recovery 2021-04-16 23:14:43,109 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4651] Performing scope/collection specific operations 2021-04-16 23:14:43,115 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4741] Done Performing scope/collection specific operations 2021-04-16 23:14:43,153 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:301] Starting rebalance operation of type : graceful_failover_recovery 2021-04-16 23:14:43,154 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:592] failing over nodes [ip:172.23.136.253 port:8091 ssh_username:Administrator, ip:172.23.136.254 port:8091 ssh_username:Administrator] 2021-04-16 23:14:50,756 | test | INFO | pool-2-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *................................................................................*....................X | 10642.237 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-04-16 23:14:51,994 | test | INFO | pool-2-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*X | 496.181 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-04-16 23:14:52,256 | test | INFO | pool-2-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................*................................................................X | 1052.651 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-04-16 23:22:00,799 | test | ERROR | pool-2-thread-4 | [rest_client:_http_request:748] GET http://172.23.136.128:8091/pools/default/tasks body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password Build was aborted Aborted by Balakumaran G Archiving artifacts Recording test results ERROR: Step ‘Publish JUnit test result report’ failed: None of the test reports contained any result [description-setter] Description set: [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: ABORTED