Started by remote host 172.23.107.166 [EnvInject] - Loading node environment variables. Building remotely on slv-s63703-32g-12c (P0 jython_slave bkrs3) in workspace /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0 [WS-CLEANUP] Deleting project workspace... Running Prebuild steps [centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0] $ /bin/sh -xe /tmp/jenkins1516241562676517408.sh ++ echo collections-step_wise_failover_and_recovery_P0-Jul-12-19:16:27-7.0.0-2588 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc='collections step_wise_failover_and_recovery_P0' + echo Desc: 7.0.0-2588 - collections step_wise_failover_and_recovery_P0 - centos Desc: 7.0.0-2588 - collections step_wise_failover_and_recovery_P0 - centos + echo newState=available + newState=available Success build forhudson.tasks.Shell@6fea06cb [description-setter] Description set: 7.0.0-2588 - collections step_wise_failover_and_recovery_P0 - centos Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@34f1079e [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Success build fororg.jenkinsci.plugins.envinject.EnvInjectBuilder@5a88b99d Cloning the remote Git repository Using shallow clone Cloning repository git://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0 # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 6e99e76f401c9514922fdc6e5de15098f226e419 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 6e99e76f401c9514922fdc6e5de15098f226e419 > /usr/bin/git rev-list 6e99e76f401c9514922fdc6e5de15098f226e419 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #42873 jenkins-test_suite_executor-TAF-42873 # timeout=10 No emails were triggered. [centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0] $ /bin/sh -xe /tmp/jenkins5079415316047935663.sh + echo Desc: collections-step_wise_failover_and_recovery_P0-Jul-12-19:16:27-7.0.0-2588 Desc: collections-step_wise_failover_and_recovery_P0-Jul-12-19:16:27-7.0.0-2588 [description-setter] Could not determine description. [centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0] $ /bin/sh -xe /tmp/jenkins836278500400994296.sh ++ find /root/.gradle/ -name '*.out.log' + for file in '`find ~/.gradle/ -name "*.out.log"`' ++ lsof /root/.gradle/daemon/6.4/daemon-16754.out.log ++ grep -v COMMAND ++ wc -l + lsof_line_count=1 + '[' 1 -eq 0 ']' + for file in '`find ~/.gradle/ -name "*.out.log"`' ++ lsof /root/.gradle/daemon/6.4/daemon-26452.out.log ++ wc -l ++ grep -v COMMAND + lsof_line_count=1 + '[' 1 -eq 0 ']' + for file in '`find ~/.gradle/ -name "*.out.log"`' ++ lsof /root/.gradle/daemon/6.4/daemon-31976.out.log ++ grep -v COMMAND ++ wc -l + lsof_line_count=1 + '[' 1 -eq 0 ']' + for file in '`find ~/.gradle/ -name "*.out.log"`' ++ lsof /root/.gradle/daemon/6.4/daemon-3326.out.log ++ grep -v COMMAND ++ wc -l + lsof_line_count=1 + '[' 1 -eq 0 ']' + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + [[ 7.0.0-2588 > 6.5 ]] + rerun_job=true + touch rerun_props_file + '[' false == false ']' + '[' true == true ']' + guides/gradlew --refresh-dependencies --stacktrace rerun_job -P jython=/opt/jython/bin/jython -P 'args=7.0.0-2588 --executor_jenkins_job --manual_run' Starting a Gradle Daemon, 59 busy Daemons could not be reused, use --status for details > Configure project : Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py 7.0.0-2588 --executor_jenkins_job --manual_run Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py 7.0.0-2588 --executor_jenkins_job --manual_run Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py 7.0.0-2588 --executor_jenkins_job --manual_run Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py 7.0.0-2588 --executor_jenkins_job --manual_run > Task :compileJava Note: /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :rerun_job reading from centos_collections_step_wise_failover_and_recovery_P0_7.0.0-2588 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 33s 2 actionable tasks: 2 executed [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'rerun_props_file' [EnvInject] - Variables injected successfully. [centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0] $ /bin/sh -xe /tmp/jenkins5558757385550461774.sh + sed -i 's/ALLOW_HTP.*/ALLOW_HTP = False/' lib/testconstants.py + ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 95681 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 202400 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 95681 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited + jython_path=/opt/jython/bin/jython + [[ 7.0.0-2588 > 6.5 ]] + rerun_job=true + set +e + echo newState=available + newState=available ++ echo 7.0.0-2588 ++ awk '{print substr($0,1,1)}' + majorRelease=7 + echo the major release is 7 the major release is 7 + echo '"172.23.108.210","172.23.108.211","172.23.121.216","172.23.122.63","172.23.122.64"' "172.23.108.210","172.23.108.211","172.23.121.216","172.23.122.63","172.23.122.64" + python -m easy_install httplib2 Searching for httplib2 Best match: httplib2 0.9.2 Adding httplib2 0.9.2 to easy-install.pth file Using /usr/lib/python2.7/site-packages Processing dependencies for httplib2 Finished processing dependencies for httplib2 + python scripts/populateIni.py -s '"172.23.108.210","172.23.108.211","172.23.121.216","172.23.122.63","172.23.122.64"' -d None -a None -i b/resources/5-nodes-template.ini -p centos -o /tmp/testexec.9652.ini in main the ini file is b/resources/5-nodes-template.ini the server info is ["172.23.108.210","172.23.108.211","172.23.121.216","172.23.122.63","172.23.122.64"] [global] username:root password:couchbase data_path:/data [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 [_1] ip:172.23.108.210 port:8091 n1ql_port:18093 index_port:9102 [_2] ip:172.23.108.211 port:8091 [_3] ip:172.23.121.216 port:8091 [_4] ip:172.23.122.63 port:8091 [_5] ip:172.23.122.64 port:8091 + '[' centos = windows ']' + parallel=true + '[' None = None ']' + extraInstall= + echo extra install is extra install is + '[' GROUP=P0_step_wise_failover_and_recovery,rerun=False = '' ']' + parameters=GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 + '[' centos = windows ']' + guides/gradlew --refresh-dependencies iptables -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.9652.ini iptables -F' > Configure project : Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/testexec.9652.ini iptables -F Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/testexec.9652.ini iptables -F Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/testexec.9652.ini iptables -F Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/testexec.9652.ini iptables -F > Task :iptables ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2 172.23.122.63 172.23.122.64 172.23.121.216 172.23.108.211 172.23.108.210 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 12s 1 actionable task: 1 executed + mkdir tr_for_install + cd tr_for_install + git clone https://github.com/couchbase/testrunner.git Cloning into 'testrunner'... Checking out files: 44% (1183/2661) Checking out files: 45% (1198/2661) Checking out files: 46% (1225/2661) Checking out files: 47% (1251/2661) Checking out files: 48% (1278/2661) Checking out files: 49% (1304/2661) Checking out files: 50% (1331/2661) Checking out files: 51% (1358/2661) Checking out files: 52% (1384/2661) Checking out files: 53% (1411/2661) Checking out files: 54% (1437/2661) Checking out files: 55% (1464/2661) Checking out files: 56% (1491/2661) Checking out files: 57% (1517/2661) Checking out files: 58% (1544/2661) Checking out files: 59% (1570/2661) Checking out files: 60% (1597/2661) Checking out files: 61% (1624/2661) Checking out files: 62% (1650/2661) Checking out files: 63% (1677/2661) Checking out files: 64% (1704/2661) Checking out files: 65% (1730/2661) Checking out files: 66% (1757/2661) Checking out files: 67% (1783/2661) Checking out files: 68% (1810/2661) Checking out files: 69% (1837/2661) Checking out files: 70% (1863/2661) Checking out files: 71% (1890/2661) Checking out files: 72% (1916/2661) Checking out files: 73% (1943/2661) Checking out files: 74% (1970/2661) Checking out files: 75% (1996/2661) Checking out files: 76% (2023/2661) Checking out files: 77% (2049/2661) Checking out files: 78% (2076/2661) Checking out files: 79% (2103/2661) Checking out files: 80% (2129/2661) Checking out files: 81% (2156/2661) Checking out files: 82% (2183/2661) Checking out files: 83% (2209/2661) Checking out files: 84% (2236/2661) Checking out files: 85% (2262/2661) Checking out files: 86% (2289/2661) Checking out files: 87% (2316/2661) Checking out files: 88% (2342/2661) Checking out files: 89% (2369/2661) Checking out files: 90% (2395/2661) Checking out files: 91% (2422/2661) Checking out files: 92% (2449/2661) Checking out files: 93% (2475/2661) Checking out files: 94% (2502/2661) Checking out files: 95% (2528/2661) Checking out files: 96% (2555/2661) Checking out files: 97% (2582/2661) Checking out files: 98% (2608/2661) Checking out files: 99% (2635/2661) Checking out files: 100% (2661/2661) Checking out files: 100% (2661/2661), done. + cd testrunner + python3 scripts/new_install.py -i /tmp/testexec.9652.ini -p timeout=1200,get-cbcollect-info=True,version=7.0.0-2588,product=cb,debug_logs=False,url= 2020-07-12 19:17:46,420 - root - WARNING - URL: is not valid, will use version to locate build 2020-07-12 19:17:46,421 - root - INFO - SSH Connecting to 172.23.108.210 with username:root, attempt#1 of 5 2020-07-12 19:17:46,616 - root - INFO - SSH Connected to 172.23.108.210 as root 2020-07-12 19:17:46,848 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:47,350 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:47,351 - root - INFO - SSH Connecting to 172.23.108.211 with username:root, attempt#1 of 5 2020-07-12 19:17:47,651 - root - INFO - SSH Connected to 172.23.108.211 as root 2020-07-12 19:17:47,921 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:48,348 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:48,350 - root - INFO - SSH Connecting to 172.23.121.216 with username:root, attempt#1 of 5 2020-07-12 19:17:48,640 - root - INFO - SSH Connected to 172.23.121.216 as root 2020-07-12 19:17:49,002 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:49,356 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:49,358 - root - INFO - SSH Connecting to 172.23.122.63 with username:root, attempt#1 of 5 2020-07-12 19:17:49,540 - root - INFO - SSH Connected to 172.23.122.63 as root 2020-07-12 19:17:49,815 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:50,111 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:50,113 - root - INFO - SSH Connecting to 172.23.122.64 with username:root, attempt#1 of 5 2020-07-12 19:17:50,214 - root - INFO - SSH Connected to 172.23.122.64 as root 2020-07-12 19:17:50,503 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:50,801 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:50,803 - root - INFO - SSH Connecting to 172.23.108.210 with username:root, attempt#1 of 5 2020-07-12 19:17:50,937 - root - INFO - SSH Connected to 172.23.108.210 as root 2020-07-12 19:17:51,178 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:51,578 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:51,580 - root - INFO - SSH Connecting to 172.23.108.211 with username:root, attempt#1 of 5 2020-07-12 19:17:51,734 - root - INFO - SSH Connected to 172.23.108.211 as root 2020-07-12 19:17:51,948 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:52,288 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:52,290 - root - INFO - SSH Connecting to 172.23.121.216 with username:root, attempt#1 of 5 2020-07-12 19:17:52,394 - root - INFO - SSH Connected to 172.23.121.216 as root 2020-07-12 19:17:52,671 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:52,974 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:52,976 - root - INFO - SSH Connecting to 172.23.122.63 with username:root, attempt#1 of 5 2020-07-12 19:17:53,076 - root - INFO - SSH Connected to 172.23.122.63 as root 2020-07-12 19:17:53,336 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:53,634 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:53,636 - root - INFO - SSH Connecting to 172.23.122.64 with username:root, attempt#1 of 5 2020-07-12 19:17:53,734 - root - INFO - SSH Connected to 172.23.122.64 as root 2020-07-12 19:17:53,997 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2020-07-12 19:17:54,292 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2020-07-12 19:17:54,293 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm 2020-07-12 19:17:54,296 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm is live 2020-07-12 19:17:54,296 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm 2020-07-12 19:17:54,297 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm is live 2020-07-12 19:17:54,297 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm 2020-07-12 19:17:54,299 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm is live 2020-07-12 19:17:54,299 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm 2020-07-12 19:17:54,300 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm is live 2020-07-12 19:17:54,300 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm 2020-07-12 19:17:54,301 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/2588/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm is live 2020-07-12 19:17:54,302 - root - INFO - Downloading build binary to 172.23.108.210:/tmp/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm.. 2020-07-12 19:17:54,408 - root - INFO - Downloading build binary to 172.23.108.211:/tmp/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm.. 2020-07-12 19:17:54,505 - root - INFO - Downloading build binary to 172.23.121.216:/tmp/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm.. 2020-07-12 19:17:54,585 - root - INFO - Downloading build binary to 172.23.122.63:/tmp/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm.. 2020-07-12 19:17:54,659 - root - INFO - Downloading build binary to 172.23.122.64:/tmp/couchbase-server-enterprise-7.0.0-2588-centos7.x86_64.rpm.. 2020-07-12 19:18:28,784 - root - INFO - Done with uninstall on 172.23.122.63. 2020-07-12 19:18:28,884 - root - INFO - Done with uninstall on 172.23.122.64. 2020-07-12 19:18:29,100 - root - INFO - Done with uninstall on 172.23.121.216. 2020-07-12 19:18:29,521 - root - INFO - Done with uninstall on 172.23.108.210. 2020-07-12 19:18:29,697 - root - INFO - Done with uninstall on 172.23.108.211. 2020-07-12 19:19:27,822 - root - INFO - Done with install on 172.23.122.63. 2020-07-12 19:19:28,664 - root - INFO - Done with install on 172.23.121.216. 2020-07-12 19:19:28,884 - root - INFO - Done with install on 172.23.122.64. 2020-07-12 19:20:02,241 - root - INFO - Done with install on 172.23.108.210. 2020-07-12 19:20:05,575 - root - INFO - Done with install on 172.23.108.211. 2020-07-12 19:20:27,882 - root - INFO - running command.raw on 172.23.122.63: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.122.63 -u Administrator -p password 2020-07-12 19:20:28,156 - root - INFO - command executed successfully with root 2020-07-12 19:20:28,162 - root - INFO - running command.raw on 172.23.122.63: rm -rf /data/* 2020-07-12 19:20:28,190 - root - INFO - command executed successfully with root 2020-07-12 19:20:28,191 - root - INFO - running command.raw on 172.23.122.63: chown -R couchbase:couchbase /data 2020-07-12 19:20:28,247 - root - INFO - command executed successfully with root 2020-07-12 19:20:28,248 - root - INFO - /nodes/self/controller/settings params : path=%2Fdata 2020-07-12 19:20:28,725 - root - INFO - running command.raw on 172.23.121.216: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.121.216 -u Administrator -p password 2020-07-12 19:20:28,944 - root - INFO - running command.raw on 172.23.122.64: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.122.64 -u Administrator -p password 2020-07-12 19:20:28,975 - root - INFO - command executed successfully with root 2020-07-12 19:20:28,982 - root - INFO - running command.raw on 172.23.121.216: rm -rf /data/* 2020-07-12 19:20:28,996 - root - INFO - command executed successfully with root 2020-07-12 19:20:28,997 - root - INFO - running command.raw on 172.23.121.216: chown -R couchbase:couchbase /data 2020-07-12 19:20:29,065 - root - INFO - command executed successfully with root 2020-07-12 19:20:29,065 - root - INFO - /nodes/self/controller/settings params : path=%2Fdata 2020-07-12 19:20:29,193 - root - INFO - command executed successfully with root 2020-07-12 19:20:29,200 - root - INFO - running command.raw on 172.23.122.64: rm -rf /data/* 2020-07-12 19:20:29,220 - root - INFO - command executed successfully with root 2020-07-12 19:20:29,220 - root - INFO - running command.raw on 172.23.122.64: chown -R couchbase:couchbase /data 2020-07-12 19:20:29,280 - root - INFO - command executed successfully with root 2020-07-12 19:20:29,280 - root - INFO - /nodes/self/controller/settings params : path=%2Fdata 2020-07-12 19:20:29,720 - root - INFO - Setting data_path: /data: status True 2020-07-12 19:20:30,491 - root - INFO - Setting data_path: /data: status True 2020-07-12 19:20:30,700 - root - INFO - Setting data_path: /data: status True 2020-07-12 19:20:30,732 - root - INFO - Setting KV memory quota as 2684 MB on 172.23.122.63 2020-07-12 19:20:30,732 - root - INFO - pools/default params : memoryQuota=2684 2020-07-12 19:20:30,738 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2020-07-12 19:20:30,739 - root - INFO - /node/controller/setupServices params on 172.23.122.63: 8091:hostname=None&user=Administrator&password=password&services=kv 2020-07-12 19:20:30,745 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-12 19:20:30,745 - root - INFO - settings/web params on 172.23.122.63:8091:port=8091&username=Administrator&password=password 2020-07-12 19:20:30,785 - root - INFO - --> status:True 2020-07-12 19:20:30,785 - root - INFO - Done with init on 172.23.122.63. 2020-07-12 19:20:30,823 - root - INFO - Done with cleanup on 172.23.122.63. 2020-07-12 19:20:31,495 - root - INFO - Setting KV memory quota as 2684 MB on 172.23.121.216 2020-07-12 19:20:31,496 - root - INFO - pools/default params : memoryQuota=2684 2020-07-12 19:20:31,509 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2020-07-12 19:20:31,510 - root - INFO - /node/controller/setupServices params on 172.23.121.216: 8091:hostname=None&user=Administrator&password=password&services=kv 2020-07-12 19:20:31,541 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-12 19:20:31,541 - root - INFO - settings/web params on 172.23.121.216:8091:port=8091&username=Administrator&password=password 2020-07-12 19:20:31,583 - root - INFO - --> status:True 2020-07-12 19:20:31,583 - root - INFO - Done with init on 172.23.121.216. 2020-07-12 19:20:31,613 - root - INFO - Done with cleanup on 172.23.121.216. 2020-07-12 19:20:31,704 - root - INFO - Setting KV memory quota as 2684 MB on 172.23.122.64 2020-07-12 19:20:31,705 - root - INFO - pools/default params : memoryQuota=2684 2020-07-12 19:20:31,712 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2020-07-12 19:20:31,712 - root - INFO - /node/controller/setupServices params on 172.23.122.64: 8091:hostname=None&user=Administrator&password=password&services=kv 2020-07-12 19:20:31,719 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-12 19:20:31,719 - root - INFO - settings/web params on 172.23.122.64:8091:port=8091&username=Administrator&password=password 2020-07-12 19:20:31,760 - root - INFO - --> status:True 2020-07-12 19:20:31,760 - root - INFO - Done with init on 172.23.122.64. 2020-07-12 19:20:31,788 - root - INFO - Done with cleanup on 172.23.122.64. 2020-07-12 19:21:02,300 - root - INFO - running command.raw on 172.23.108.210: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.108.210 -u Administrator -p password 2020-07-12 19:21:02,701 - root - INFO - command executed successfully with root 2020-07-12 19:21:02,710 - root - INFO - running command.raw on 172.23.108.210: rm -rf /data/* 2020-07-12 19:21:02,725 - root - INFO - command executed successfully with root 2020-07-12 19:21:02,726 - root - INFO - running command.raw on 172.23.108.210: chown -R couchbase:couchbase /data 2020-07-12 19:21:02,787 - root - INFO - command executed successfully with root 2020-07-12 19:21:02,787 - root - INFO - /nodes/self/controller/settings params : path=%2Fdata 2020-07-12 19:21:05,468 - root - INFO - Setting data_path: /data: status True 2020-07-12 19:21:05,634 - root - INFO - running command.raw on 172.23.108.211: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.108.211 -u Administrator -p password 2020-07-12 19:21:06,054 - root - INFO - command executed successfully with root 2020-07-12 19:21:06,063 - root - INFO - running command.raw on 172.23.108.211: rm -rf /data/* 2020-07-12 19:21:06,082 - root - INFO - command executed successfully with root 2020-07-12 19:21:06,082 - root - INFO - running command.raw on 172.23.108.211: chown -R couchbase:couchbase /data 2020-07-12 19:21:06,150 - root - INFO - command executed successfully with root 2020-07-12 19:21:06,150 - root - INFO - /nodes/self/controller/settings params : path=%2Fdata 2020-07-12 19:21:06,474 - root - INFO - Setting KV memory quota as 2553 MB on 172.23.108.210 2020-07-12 19:21:06,474 - root - INFO - pools/default params : memoryQuota=2553 2020-07-12 19:21:06,487 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2020-07-12 19:21:06,488 - root - INFO - /node/controller/setupServices params on 172.23.108.210: 8091:hostname=None&user=Administrator&password=password&services=kv 2020-07-12 19:21:06,501 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-12 19:21:06,502 - root - INFO - settings/web params on 172.23.108.210:8091:port=8091&username=Administrator&password=password 2020-07-12 19:21:06,617 - root - INFO - --> status:True 2020-07-12 19:21:06,617 - root - INFO - Done with init on 172.23.108.210. 2020-07-12 19:21:06,654 - root - INFO - Done with cleanup on 172.23.108.210. 2020-07-12 19:21:08,817 - root - INFO - Setting data_path: /data: status True 2020-07-12 19:21:09,823 - root - INFO - Setting KV memory quota as 2553 MB on 172.23.108.211 2020-07-12 19:21:09,823 - root - INFO - pools/default params : memoryQuota=2553 2020-07-12 19:21:09,839 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2020-07-12 19:21:09,839 - root - INFO - /node/controller/setupServices params on 172.23.108.211: 8091:hostname=None&user=Administrator&password=password&services=kv 2020-07-12 19:21:09,853 - root - INFO - --> in init_cluster...Administrator,password,8091 2020-07-12 19:21:09,853 - root - INFO - settings/web params on 172.23.108.211:8091:port=8091&username=Administrator&password=password 2020-07-12 19:21:09,950 - root - INFO - --> status:True 2020-07-12 19:21:09,950 - root - INFO - Done with init on 172.23.108.211. 2020-07-12 19:21:09,976 - root - INFO - Done with cleanup on 172.23.108.211. 2020-07-12 19:21:14,933 - root - INFO - ---------------------------------------------------------------------------------------------------- 2020-07-12 19:21:14,943 - root - INFO - node:172.23.108.210:8091 version:7.0.0-2588-enterprise aFamily:inet services:['kv'] 2020-07-12 19:21:14,951 - root - INFO - node:172.23.108.211:8091 version:7.0.0-2588-enterprise aFamily:inet services:['kv'] 2020-07-12 19:21:14,958 - root - INFO - node:172.23.121.216:8091 version:7.0.0-2588-enterprise aFamily:inet services:['kv'] 2020-07-12 19:21:14,964 - root - INFO - node:172.23.122.63:8091 version:7.0.0-2588-enterprise aFamily:inet services:['kv'] 2020-07-12 19:21:14,974 - root - INFO - node:172.23.122.64:8091 version:7.0.0-2588-enterprise aFamily:inet services:['kv'] 2020-07-12 19:21:14,974 - root - INFO - ---------------------------------------------------------------------------------------------------- 2020-07-12 19:21:14,974 - root - INFO - ---------------------------------------------------------------------------------------------------- 2020-07-12 19:21:14,974 - root - INFO - INSTALL COMPLETED ON: 172.23.108.210 2020-07-12 19:21:14,974 - root - INFO - INSTALL COMPLETED ON: 172.23.108.211 2020-07-12 19:21:14,975 - root - INFO - INSTALL COMPLETED ON: 172.23.121.216 2020-07-12 19:21:14,975 - root - INFO - INSTALL COMPLETED ON: 172.23.122.63 2020-07-12 19:21:14,975 - root - INFO - INSTALL COMPLETED ON: 172.23.122.64 2020-07-12 19:21:14,975 - root - INFO - ---------------------------------------------------------------------------------------------------- 2020-07-12 19:21:14,975 - root - INFO - TOTAL INSTALL TIME = 209 seconds + status=0 + cd ../.. + '[' 0 -eq 0 ']' ++ echo collections-step_wise_failover_and_recovery_P0-Jul-12-19:16:27-7.0.0-2588 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc2='collections step_wise_failover_and_recovery_P0' + '[' 7 = 3 ']' + [[ master != \m\a\s\t\e\r ]] + echo 'Need to set ALLOW_HTP back to True to do git pull branch' Need to set ALLOW_HTP back to True to do git pull branch + sed -i 's/ALLOW_HTP.*/ALLOW_HTP = True/' lib/testconstants.py + git checkout master Switched to a new branch 'master' Branch master set up to track remote branch master from origin. + git pull origin master From git://github.com/couchbaselabs/TAF * branch master -> FETCH_HEAD Already up-to-date. + echo 'Set ALLOW_HTP to False so test could run.' Set ALLOW_HTP to False so test could run. + sed -i 's/ALLOW_HTP.*/ALLOW_HTP = False/' lib/testconstants.py + git fetch + git reset --hard origin/master HEAD is now at 6e99e76 Fixing a typo + '[' '' '!=' None ']' + sh -c '' + find /data/workspace/centos8-p0-os_certify-vset00-00-collections/logs/testrunner-20-Jun-29_17-59-28 /data/workspace/centos8-p0-os_certify-vset00-00-ent-backup-recovery/logs/testrunner-20-Jul-09_23-40-25 /data/workspace/centos8-p0-os_certify-vset00-00-transaction/logs/testrunner-20-Jul-02_17-41-00 /data/workspace/centosnonroot-p0-os_certify-vset00-00-ent-backup-recovery/logs/testrunner-20-Jul-11_19-55-09 /data/workspace/centosnonroot-p0-os_certify-vset00-00-nsserver/logs/testrunner-20-Jul-03_10-55-43 /data/workspace/centos-nserv-autofailover-machine-restart/logs/testrunner-20-Jul-10_03-30-30 /data/workspace/centos-nserv-autofailover-machine-restart/logs/testrunner-20-Jul-12_16-10-40 /data/workspace/centos-p0-2i-vset00-00-add-based-recovery-dgm-plasma-P0/logs/testrunner-20-Jul-02_21-56-44 /data/workspace/centos-p0-2i-vset00-00-basic-integration-memdb/logs/testrunner-20-Jun-26_05-00-22 /data/workspace/centos-p0-2i-vset00-00-bootstrap-plasma/logs/testrunner-20-Jun-26_03-59-28 /data/workspace/centos-p0-2i-vset00-00-bucket-ops/logs/testrunner-20-Jun-28_13-43-05 /data/workspace/centos-p0-2i-vset00-00-ci-tests-memdb/logs/testrunner-20-Jun-23_01-43-38 /data/workspace/centos-p0-2i-vset00-00-create-scan-drop-multi-bucket-plasma/logs/testrunner-20-Jun-27_11-05-25 /data/workspace/centos-p0-2i-vset00-00-deployment-plan-memdb/logs/testrunner-20-Jul-05_03-23-30 /data/workspace/centos-p0-2i-vset00-00-doc-key-size-memdb-P0/logs/testrunner-20-Jun-23_04-13-06 /data/workspace/centos-p0-2i-vset00-00-index-partitioning-clusterops-plasma/logs/testrunner-20-Jul-01_05-41-18 /data/workspace/centos-p0-2i-vset00-00-index-partitioning-scans-memdb/logs/testrunner-20-Jun-23_06-08-48 /data/workspace/centos-p0-2i-vset00-00-multiscan_with_offset_and_distinct_memdb/logs/testrunner-20-Jul-05_01-38-32 /data/workspace/centos-p0-2i-vset00-00-query-based-recovery-dgm-plasma-P1/logs/testrunner-20-Jul-05_08-32-42 /data/workspace/centos-p0-2i-vset00-00-rebalance_green_plasma/logs/testrunner-20-Jun-23_04-37-43 /data/workspace/centos-p0-2i-vset00-00-rebalance_red_memdb/logs/testrunner-20-Jun-23_04-42-47 /data/workspace/centos-p0-2i-vset00-00-recovery-add-index-memdb/logs/testrunner-20-Jul-05_08-04-49 /data/workspace/centos-p0-2i-vset00-00-rollback-recovery/logs/testrunner-20-Jul-09_22-34-07 /data/workspace/centos-p0-backup_recovery-vset00-00-merge-p1-crash/logs/testrunner-20-Jul-11_20-27-29 /data/workspace/centos-p0-backup_recovery-vset00-00-mh-enhance_6.5_P1/logs/testrunner-20-Jul-06_08-41-40 /data/workspace/centos-p0-backup_recovery-vset00-00-obj-enhance-1/logs/testrunner-20-Jul-10_00-58-46 /data/workspace/centos-p0-backup_recovery-vset00-00-obj-negative-tests/logs/testrunner-20-Jul-06_08-46-49 /data/workspace/centos-p0-backup_recovery-vset00-00-obj-purge-resume/logs/testrunner-20-Jul-06_09-41-49 /data/workspace/centos-p0-backup_recovery-vset00-00-obj-rbac/logs/testrunner-20-Jul-06_12-07-33 /data/workspace/centos-p0-backup_recovery-vset00-00-rbac/logs/testrunner-20-Jun-28_17-53-27 /data/workspace/centos-p0-backup_recovery-vset00-00-rbac-obj/logs/testrunner-20-Jul-06_05-57-59 /data/workspace/centos-p0-backup_recovery-vset00-00-regex-ssl/logs/testrunner-20-Jul-05_06-20-31 /data/workspace/centos-p0-collections-vset00-00-auto_failover_durability/logs/testrunner-20-Jun-28_17-24-16 /data/workspace/centos-p0-collections-vset00-00-auto_retry_failed_rebalance_P1/logs/testrunner-20-Jun-22_11-49-41 /data/workspace/centos-p0-collections-vset00-00-bucket_flush_P0/logs/testrunner-20-Jul-02_15-07-59 /data/workspace/centos-p0-collections-vset00-00-bucket_warmup/logs/testrunner-20-Jul-02_15-27-59 /data/workspace/centos-p0-collections-vset00-00-compaction_P0/logs/testrunner-20-Jul-07_10-51-34 /data/workspace/centos-p0-collections-vset00-00-durability_majority_success_P1/logs/JavaSDK.log /data/workspace/centos-p0-collections-vset00-00-durability_majority_success_P1/logs/testrunner-20-Jul-12_18-28-01 /data/workspace/centos-p0-collections-vset00-00-durability_majority_success_P1/logs/test_suite_executor-TAF_42417__testresult.xml /data/workspace/centos-p0-collections-vset00-00-failover_and_rebalance_out_P0/logs/testrunner-20-Jul-04_01-47-54 /data/workspace/centos-p0-collections-vset00-00-failover_and_recovery_P0/logs/JavaSDK.log /data/workspace/centos-p0-collections-vset00-00-failover_and_recovery_P0/logs/testrunner-20-Jul-12_18-36-09 /data/workspace/centos-p0-collections-vset00-00-failover_and_recovery_P0/logs/test_suite_executor-TAF_42465__testresult.xml /data/workspace/centos-p0-collections-vset00-00-failover_and_recovery_P1/logs/testrunner-20-Jun-28_15-18-47 /data/workspace/centos-p0-collections-vset00-00-max_ttl/logs/testrunner-20-Jul-11_17-32-02 /data/workspace/centos-p0-collections-vset00-00-process_crash_P1/logs/JavaSDK.log /data/workspace/centos-p0-collections-vset00-00-process_crash_P1/logs/testrunner-20-Jun-28_20-46-39 /data/workspace/centos-p0-collections-vset00-00-process_crash_P1/logs/test_suite_executor-TAF_39088__testresult.xml /data/workspace/centos-p0-collections-vset00-00-rebalance_bucket_compaction/logs/testrunner-20-Jul-11_18-01-15 /data/workspace/centos-p0-collections-vset00-00-rebalance_collections_maxt_ttl_durability_majority_and_persist_active/logs/testrunner-20-Jul-11_19-44-34 /data/workspace/centos-p0-collections-vset00-00-rebalance_durability_majority/logs/testrunner-20-Jul-04_03-42-39 /data/workspace/centos-p0-collections-vset00-00-sanity_P0/logs/testrunner-20-Jul-12_18-30-41 /data/workspace/centos-p0-collections-vset00-00-steady_state_P1/logs/testrunner-20-Jul-04_18-59-46 /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_6.5_P0_majority/logs/testrunner-20-Jul-11_11-39-02 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_6.5_P0_persist_active/logs/testrunner-20-Jul-10_23-30-30 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_6.5_P1_majority/logs/testrunner-20-Jun-30_22-37-40 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_memcached_P0_persist_majority/logs/testrunner-20-Jun-27_08-52-11 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_multi_node_P0_persist_active/logs/testrunner-20-Jul-02_17-06-29 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_multi_node_P0_persist_majority/logs/testrunner-20-Jul-06_05-45-35 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_server_group_P0_persist_active/logs/testrunner-20-Jul-11_13-12-11 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_server_stop_start_P0_majority/logs/testrunner-20-Jul-05_03-31-45 /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_server_stop_start_P0_persist_majority/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-auto_fail_over_server_stop_start_P0_persist_majority/logs/testrunner-20-Jul-02_22-02-42 /data/workspace/centos-p0-durability-vset00-00-basic_castest_6.5_ephemeral_P0_majority/logs/testrunner-20-Jul-09_11-52-11 /data/workspace/centos-p0-durability-vset00-00-basic_castest_6.5_P0_majority/logs/testrunner-20-Jul-08_17-51-42 /data/workspace/centos-p0-durability-vset00-00-basic_castest_6.5_P0_persist_active/logs/testrunner-20-Jul-09_22-12-52 /data/workspace/centos-p0-durability-vset00-00-basic_castest_6.5_P0_persist_majority/logs/testrunner-20-Jul-04_23-32-15 /data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P0_majority/logs/testrunner-20-Jun-26_01-17-24 /data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P0_persist_active/logs/testrunner-20-Jul-05_00-29-01 /data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P0_persist_majority/logs/testrunner-20-Jul-06_19-49-52 /data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P1_majority/logs/testrunner-20-Jul-04_23-59-56 /data/workspace/centos-p0-durability-vset00-00-bucket_compaction_6.5_P1_persist_majority/logs/testrunner-20-Jul-11_13-14-57 /data/workspace/centos-p0-durability-vset00-00-bucket_durability_6.6_P1/logs/testrunner-20-Jun-22_23-02-06 /data/workspace/centos-p0-durability-vset00-00-castest_6.5_P1_majority/logs/testrunner-20-Jun-26_02-54-24 /data/workspace/centos-p0-durability-vset00-00-fail_over_6.5_P1_persist_majority/logs/testrunner-20-Jul-10_02-27-04 /data/workspace/centos-p0-durability-vset00-00-fail_over_P1/logs/testrunner-20-Jul-10_05-32-27 /data/workspace/centos-p0-durability-vset00-00-maxttl_6.5_ephemeral_P1_majority/logs/testrunner-20-Jul-11_18-07-31 /data/workspace/centos-p0-durability-vset00-00-maxttl_6.5_P0_majority/logs/testrunner-20-Jun-22_09-08-03 /data/workspace/centos-p0-durability-vset00-00-maxttl_6.5_P0_persist_active/logs/testrunner-20-Jul-10_01-49-04 /data/workspace/centos-p0-durability-vset00-00-maxttl_6.5_P1_majority/logs/testrunner-20-Jul-12_15-48-36 /data/workspace/centos-p0-durability-vset00-00-maxttl_6.5_P1_persist_active/logs/testrunner-20-Jul-12_15-49-28 /data/workspace/centos-p0-durability-vset00-00-negative_fail_over_6.5_P0_majority/logs/testrunner-20-Jul-10_02-55-02 /data/workspace/centos-p0-durability-vset00-00-process_crash_6.5_P0_persist_active/logs/testrunner-20-Jul-11_18-31-04 /data/workspace/centos-p0-durability-vset00-00-rebalance_bucket_durability_6.6_majority/logs/testrunner-20-Jul-12_17-11-03 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_ephemeral_majority_6.5_P0/logs/testrunner-20-Jun-28_20-25-30 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_majority_6.5_P0/logs/testrunner-20-Jul-04_05-17-30 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_majority_6.5_P1/logs/testrunner-20-Jul-12_01-51-02 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_out_ephemeral_majority_6.5_P0/logs/testrunner-20-Jul-06_00-51-03 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_out_majority_6.5_P0/logs/testrunner-20-Jun-27_15-07-51 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_out_majority_6.5_P1/logs/testrunner-20-Jun-25_14-27-30 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_out_none_6.5_P0/logs/testrunner-20-Jul-09_00-36-03 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_out_persist_active_6.5_P0/logs/testrunner-20-Jul-04_07-40-33 /data/workspace/centos-p0-durability-vset00-00-rebalance_in_persist_majority_6.5_P0/logs/testrunner-20-Jul-03_07-52-09 /data/workspace/centos-p0-durability-vset00-00-rebalance_out_durabilitybreaks_rebalance_in_persist_active_6.5_P1/logs/testrunner-20-Jul-12_07-45-46 /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_ephemeral_majority_6.5_P0/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_ephemeral_majority_6.5_P0/logs/testrunner-20-Jun-26_23-54-43 /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_majority_6.5_P0/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_majority_6.5_P0/logs/testrunner-20-Jun-25_14-33-38 /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_persist_majority_6.5_P0/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_persist_majority_6.5_P0/logs/testrunner-20-Jul-04_07-49-01 /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_persist_majority_6.5_P0/logs/test_suite_executor-TAF_40793__testresult.xml /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_persist_majority_6.5_P1/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-rebalance_swap_persist_majority_6.5_P1/logs/testrunner-20-Jun-24_21-31-37 /data/workspace/centos-p0-durability-vset00-00-replica_update_6.5_ephemeral_P1_majority/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-replica_update_6.5_ephemeral_P1_majority/logs/testrunner-20-Jun-25_13-55-42 /data/workspace/centos-p0-durability-vset00-00-replica_update_6.5_ephemeral_P1_majority/logs/test_suite_executor-TAF_38301__testresult.xml /data/workspace/centos-p0-durability-vset00-00-replica_update_6.5_P0_persist_majority/logs/testrunner-20-Jun-26_09-17-14 /data/workspace/centos-p0-durability-vset00-00-replica_update_6.5_P1_majority/logs/testrunner-20-Jul-10_03-34-56 /data/workspace/centos-p0-durability-vset00-00-sdk_timeouts_6.5_ephemeral_P1_majority/logs/testrunner-20-Jul-01_06-36-27 /data/workspace/centos-p0-durability-vset00-00-sdk_timeouts_6.5_P0_majority/logs/testrunner-20-Jun-30_13-08-27 /data/workspace/centos-p0-durability-vset00-00-sdk_timeouts_6.5_P1_persist_active/logs/testrunner-20-Jul-01_06-29-13 /data/workspace/centos-p0-durability-vset00-00-sdk_timeouts_6.5_P1_persist_majority/logs/testrunner-20-Jul-04_04-17-37 /data/workspace/centos-p0-durability-vset00-00-service_cbas_6.5_P0_majority/logs/testrunner-20-Jun-23_21-20-28 /data/workspace/centos-p0-durability-vset00-00-service_cbas_6.5_P1_majority/logs/testrunner-20-Jun-26_23-29-40 /data/workspace/centos-p0-durability-vset00-00-service_cbas_6.5_P1_majority/logs/testrunner-20-Jun-26_23-29-41 /data/workspace/centos-p0-durability-vset00-00-steady_state_6.5_P0_persist_active/logs/testrunner-20-Jul-11_22-32-34 /data/workspace/centos-p0-durability-vset00-00-steady_state_dgm_6.5_P0_persist_active/logs/testrunner-20-Jul-02_23-36-13 /data/workspace/centos-p0-durability-vset00-00-steady_state_dgm_6.5_P1_majority/logs/testrunner-20-Jun-27_10-30-03 /data/workspace/centos-p0-durability-vset00-00-steady_state_doc_keys_6.5_P0_persist_active/logs/testrunner-20-Jul-10_00-49-19 /data/workspace/centos-p0-durability-vset00-00-steady_state_large_docs_6.5_ephemeral_P1_majority/logs/testrunner-20-Jul-01_02-12-24 /data/workspace/centos-p0-durability-vset00-00-steady_state_subdoc_6.5_P0_majority/logs/testrunner-20-Jul-11_11-55-13 /data/workspace/centos-p0-durability-vset00-00-steady_state_subdoc_6.5_P0_persist_majority/logs/testrunner-20-Jul-11_23-17-36 /data/workspace/centos-p0-durability-vset00-00-syncwrite_failures_6.5_ephemeral_P0_majority/logs/testrunner-20-Jul-11_00-00-15 /data/workspace/centos-p0-durability-vset00-00-syncwrite_failures_6.5_P0_persist_majority/logs/testrunner-20-Jul-01_01-44-08 /data/workspace/centos-p0-durability-vset00-00-syncwrite_failures_delete_6.5_P0_majority/logs/testrunner-20-Jul-08_19-00-23 /data/workspace/centos-p0-durability-vset00-00-syncwrite_failures_upsert_6.5_P0_persist_active/logs/testrunner-20-Jul-01_08-53-31 /data/workspace/centos-p0-durability-vset00-00-tunable_rebalance_swap_majority_6.5_P0/logs/JavaSDK.log /data/workspace/centos-p0-durability-vset00-00-tunable_rebalance_swap_majority_6.5_P0/logs/testrunner-20-Jun-24_11-36-48 /data/workspace/centos-p0-ephemeral-vset00-00-2i-deployment-plan-ephemeral/logs/testrunner-20-Jul-12_00-37-50 /data/workspace/centos-p0-eventing-vset00-00-bucket/logs/testrunner-20-Jul-05_21-30-31 /data/workspace/centos-p0-eventing-vset00-00-logging/logs/testrunner-20-Jul-01_07-14-19 /data/workspace/centos-p0-eventing-vset00-00-rebalance_bucket_op_p1/logs/testrunner-20-Jun-26_13-55-22 /data/workspace/centos-p0-eventing-vset00-00-rebalance_expiry/logs/testrunner-20-Jul-01_02-19-20 /data/workspace/centos-p0-eventing-vset00-00-rebalance_expiry_set2/logs/testrunner-20-Jul-01_02-20-10 /data/workspace/centos-p0-eventing-vset00-00-rebalance_timer_op_curl_6.5_P0/logs/testrunner-20-Jul-04_10-48-24 /data/workspace/centos-p0-fts-vset00-00-ephemeral/logs/testrunner-20-Jul-11_15-07-41 /data/workspace/centos-p0-fts-vset00-00-moving-topology-P1-scorch/logs/testrunner-20-Jul-03_00-02-02 /data/workspace/centos-p0-fts-vset00-00-querytime-scorch/logs/testrunner-20-Jul-12_12-53-53 /data/workspace/centos-p0-fts-vset00-00-sanity/logs/testrunner-20-Jul-09_22-18-43 /data/workspace/centos-p0-geo-vset00-00-sanity_01/logs/testrunner-20-Jul-05_02-25-32 /data/workspace/centos-p0-geo-vset00-00-sanity_02/logs/testrunner-20-Jun-27_09-11-52 /data/workspace/centos-p0-nserv-vset00-00-auto-failover-aborts-rebalance_6.5_P0/logs/testrunner-20-Jun-22_22-51-50 /data/workspace/centos-p0-nserv-vset00-00-swap-rebalance/logs/testrunner-20-Jul-11_17-10-09 /data/workspace/centos-p0-query-vset00-00-ansi_merge_6.5_P1/logs/testrunner-20-Jul-05_01-20-29 /data/workspace/centos-p0-query-vset00-00-array-indexing/logs/testrunner-20-Jul-02_15-20-22 /data/workspace/centos-p0-query-vset00-00-backfill/logs/testrunner-20-Jul-03_08-47-21 /data/workspace/centos-p0-query-vset00-00-bitwise-n1ql/logs/testrunner-20-Jul-05_01-24-53 /data/workspace/centos-p0-query-vset00-00-chained_let_moi_6.5_P1/logs/testrunner-20-Jul-05_00-37-14 /data/workspace/centos-p0-query-vset00-00-clusterops-2_moi/logs/testrunner-20-Jul-03_07-32-10 /data/workspace/centos-p0-query-vset00-00-collections_end2end/logs/testrunner-20-Jun-22_23-17-27 /data/workspace/centos-p0-query-vset00-00-covering-2_gsi/logs/testrunner-20-Jun-27_09-23-00 /data/workspace/centos-p0-query-vset00-00-covering-2_moi/logs/testrunner-20-Jul-05_01-17-06 /data/workspace/centos-p0-query-vset00-00-cte_moi_6.5_P1/logs/testrunner-20-Jun-22_23-33-17 /data/workspace/centos-p0-query-vset00-00-dml_moi/logs/testrunner-20-Jul-05_01-13-09 /data/workspace/centos-p0-query-vset00-00-dynamic/logs/testrunner-20-Jul-01_06-17-19 /data/workspace/centos-p0-query-vset00-00-enhancements/logs/testrunner-20-Jul-09_22-22-05 /data/workspace/centos-p0-query-vset00-00-index3_moi/logs/testrunner-20-Jul-02_16-39-12 /data/workspace/centos-p0-query-vset00-00-index_gsi/logs/testrunner-20-Jul-12_12-58-57 /data/workspace/centos-p0-query-vset00-00-joins_gsi/logs/testrunner-20-Jun-26_02-59-37 /data/workspace/centos-p0-query-vset00-00-joins_moi/logs/testrunner-20-Jul-02_19-41-54 /data/workspace/centos-p0-query-vset00-00-misc/logs/testrunner-20-Jun-23_00-32-10 /data/workspace/centos-p0-query-vset00-00-monitoring/logs/testrunner-20-Jun-26_03-05-04 /data/workspace/centos-p0-query-vset00-00-monitoring_moi/logs/testrunner-20-Jun-29_19-09-57 /data/workspace/centos-p0-query-vset00-00-prepared_gsi/logs/testrunner-20-Jun-27_11-19-11 /data/workspace/centos-p0-query-vset00-00-query_upgrade_flex/logs/testrunner-20-Jul-05_11-30-19 /data/workspace/centos-p0-query-vset00-00-rbac-n1ql/logs/testrunner-20-Jul-05_01-04-57 /data/workspace/centos-p0-query-vset00-00-rbac-special/logs/testrunner-20-Jul-05_01-09-28 /data/workspace/centos-p0-query-vset00-00-skip_range_key_scan_6.5_P1/logs/testrunner-20-Jul-05_01-28-18 /data/workspace/centos-p0-query-vset00-00-skip_range_key_scan_moi_6.5_P1/logs/testrunner-20-Jul-05_01-31-42 /data/workspace/centos-p0-query-vset00-00-stats_moi/logs/testrunner-20-Jul-03_08-54-58 /data/workspace/centos-p0-query-vset00-00-subquery/logs/testrunner-20-Jul-01_05-50-32 /data/workspace/centos-p0-query-vset00-00-tools-dml_moi/logs/testrunner-20-Jun-23_01-30-57 /data/workspace/centos-p0-query-vset00-00-tools-index_gsi/logs/testrunner-20-Jun-23_01-35-40 /data/workspace/centos-p0-query-vset00-00-tools-nondocs_gsi/logs/testrunner-20-Jul-03_04-03-25 /data/workspace/centos-p0-query-vset00-00-vset-ro/logs/testrunner-20-Jul-03_09-04-18 /data/workspace/centos-p0-rqg-vset00-00-arrays/logs/testrunner-20-Jul-05_00-03-21 /data/workspace/centos-p0-security-vset00-00-rbac-ldap-cli/logs/testrunner-20-Jun-28_00-23-41 /data/workspace/centos-p0-security-vset00-00-rbac-ldap/logs/testrunner-20-Jun-27_23-02-43 /data/workspace/centos-p0-transaction-vset00-00-autofailover_durability_majority_and_persist_on_master_6.5_P0_Set1/logs/testrunner-20-Jul-11_23-19-43 /data/workspace/centos-p0-transaction-vset00-00-crash_test_durability_persist_to_majority_6.5_P1/logs/testrunner-20-Jul-10_13-13-46 /data/workspace/centos-p0-transaction-vset00-00-diskautofailover_durability_majority_6.5_P0_Set1/logs/testrunner-20-Jul-12_11-29-53 /data/workspace/centos-p0-transaction-vset00-00-doc_isolation_durability_majority_6.5_P0/logs/testrunner-20-Jul-07_19-47-28 /data/workspace/centos-p0-transaction-vset00-00-durability_majority_and_persist_on_master_auto_retry_P0/logs/testrunner-20-Jun-28_16-34-24 /data/workspace/centos-p0-transaction-vset00-00-durability_majority_and_persist_on_master_auto_retry_P1/logs/testrunner-20-Jul-10_15-40-47 /data/workspace/centos-p0-transaction-vset00-00-durability_majority_auto_retry_P1/logs/testrunner-20-Jun-26_16-46-37 /data/workspace/centos-p0-transaction-vset00-00-multi_bucket_durability_majority_defer_6.5_P0/logs/JavaSDK.log /data/workspace/centos-p0-transaction-vset00-00-multi_bucket_durability_majority_defer_6.5_P0/logs/testrunner-20-Jul-11_10-19-26 /data/workspace/centos-p0-transaction-vset00-00-newfailover_durability_majority_6.5_P0/logs/testrunner-20-Jul-06_22-02-26 /data/workspace/centos-p0-transaction-vset00-00-rebalance_in_out_durability_majority_6.5_P0/logs/testrunner-20-Jun-30_14-13-35 /data/workspace/centos-p0-transaction-vset00-00-rebalance_in_out_durability_majority_6.5_P0_Set1/logs/testrunner-20-Jun-26_14-44-49 /data/workspace/centos-p0-transaction-vset00-00-rebalance_in_out_durability_majority_defer_6.5_P0_Set1/logs/testrunner-20-Jul-11_12-43-43 /data/workspace/centos-p0-transaction-vset00-00-rebalance_out_durability_persist_to_majority_6.5_P0_Set1/logs/testrunner-20-Jun-26_15-13-43 /data/workspace/centos-p0-transaction-vset00-00-steady_state_diff_doc_size_durability_majority_6.5_P0_Set1/logs/testrunner-20-Jul-11_23-52-55 /data/workspace/centos-p0-transaction-vset00-00-steady_state_durability_majority_6.5_P0/logs/testrunner-20-Jul-08_21-41-46 /data/workspace/centos-p0-transaction-vset00-00-steady_state_durability_majority_and_persist_on_master_6.5_P0/logs/testrunner-20-Jul-06_08-49-37 /data/workspace/centos-p0-transaction-vset00-00-steady_state_durability_persist_to_majority_6.5_P0/logs/testrunner-20-Jul-08_20-23-03 /data/workspace/centos-p0-transaction-vset00-00-steady_state_durability_persist_to_majority_defer_6.5_P0/logs/testrunner-20-Jun-22_21-48-28 /data/workspace/centos-p0-transaction-vset00-00-swap_rebalance_durability_majority_6.5_P0/logs/testrunner-20-Jul-09_01-16-50 /data/workspace/centos-p0-transaction-vset00-00-swap_rebalance_durability_majority_6.5_P0_Set1/logs/testrunner-20-Jun-28_19-21-00 /data/workspace/centos-p0-transaction-vset00-00-swap_rebalance_durability_persist_to_majority_6.5_P0_Set1/logs/testrunner-20-Jul-10_17-48-59 /data/workspace/centos-p0-xdcr-vset00-00-lww-p0-3/logs/testrunner-20-Jun-23_04-32-22 /data/workspace/centos-p0-xdcr-vset00-00-online-upgrade/logs/testrunner-20-Jun-23_04-48-21 /data/workspace/centos-p0-xdcr-vset00-00-pause-resume-collections/logs/testrunner-20-Jul-02_22-20-12 /data/workspace/centos-p0-xdcr-vset00-00-pause-resume-p1/logs/testrunner-20-Jul-11_21-49-16 /data/workspace/centos-p0-xdcr-vset00-00-rebalance-ssl-1/logs/testrunner-20-Jul-05_06-13-57 /data/workspace/centos-p0-xdcr-vset00-00-unidirectional-ssl-2/logs/testrunner-20-Jul-05_07-44-11 /data/workspace/debian10-p0-os_certify-vset00-00-n1ql/logs/testrunner-20-Jun-27_09-47-36 /data/workspace/debian10-p0-os_certify-vset00-00-transaction/logs/JavaSDK.log /data/workspace/debian10-p0-os_certify-vset00-00-transaction/logs/testrunner-20-Jun-23_23-02-21 /data/workspace/debian10-p0-os_certify-vset00-00-view/logs/testrunner-20-Jul-05_08-12-50 /data/workspace/debian10-p0-sanity-vset00-00-4-node-TAF/logs/testrunner-20-Jul-02_11-03-48 /data/workspace/mac-mojave-2node-QESanity/logs/testrunner-20-Jul-10_10-44-18 /data/workspace/oel6-4node-rebalance_in_jython/logs/JavaSDK.log /data/workspace/oel6-4node-rebalance_in_jython/logs/testrunner-20-Jun-26_03-22-22 /data/workspace/oel8-p0-os_certify-vset00-00-durability/logs/JavaSDK.log /data/workspace/oel8-p0-os_certify-vset00-00-durability/logs/testrunner-20-Jun-27_07-26-16 /data/workspace/oel8-p0-os_certify-vset00-00-ent-backup-recovery/logs/testrunner-20-Jul-03_01-03-54 /data/workspace/oel8-p0-sanity-vset00-00-4-node/logs/testrunner-20-Jul-11_16-59-38 /data/workspace/suse15-p0-os_certify-vset00-00-ent-backup-recovery/logs/testrunner-20-Jun-29_16-53-05 /data/workspace/suse15-p0-os_certify-vset00-00-transaction/logs/testrunner-20-Jul-07_01-39-22 /data/workspace/suse15-p0-sanity-vset00-00-4-node-TAF/logs/testrunner-20-Jul-08_10-55-59 /data/workspace/ubuntu20-p0-os_certify-vset00-00-durability/logs/JavaSDK.log /data/workspace/ubuntu20-p0-os_certify-vset00-00-durability/logs/testrunner-20-Jun-28_06-41-11 /data/workspace/ubuntu20-p0-os_certify-vset00-00-ent-backup-recovery/logs/testrunner-20-Jun-27_10-41-24 /data/workspace/ubuntu20-p0-os_certify-vset00-00-security-check-audit/logs/testrunner-20-Jun-27_11-25-29 /data/workspace/ubuntu20-p0-os_certify-vset00-00-transaction/logs/testrunner-20-Jul-06_20-49-13 /data/workspace/ubuntu20-p0-sanity-vset00-00-4-node-TAF/logs/testrunner-20-Jun-28_16-50-59 /data/workspace/windows-p0-backup_recovery-vset00-00-enhance-1/logs/testrunner-20-Jul-08_13-16-19 /data/workspace/windows-p0-backup_recovery-vset00-00-mh-enhance_6.5_P0/logs/testrunner-20-Jul-09_00-59-45 /data/workspace/windows-p0-durability-vset00-00-syncwrite_failures_create_6.5_windows_P0_majority/logs/testrunner-20-Jul-11_11-59-53 /data/workspace/windows-p0-os_certify-vset00-00-durability/logs/testrunner-20-Jul-12_15-32-56 /data/workspace/windows-p0-os_certify-vset00-00-ent-backup-recovery/logs/testrunner-20-Jul-08_14-11-18 /data/workspace/windows-p0-os_certify-vset00-00-eventing/logs/testrunner-20-Jul-02_16-50-22 /data/workspace/windows-p0-os_certify-vset00-00-fts/logs/testrunner-20-Jul-02_17-59-10 /data/workspace/windows-p0-os_certify-vset00-00-rza/logs/testrunner-20-Jul-12_00-34-59 /data/workspace/windows-p0-os_certify-vset00-00-transaction/logs/testrunner-20-Jun-23_03-11-16 /data/workspace/windows-p0-sanity-vset00-00-4-node/logs/testrunner-20-Jul-01_10-54-55 -type d -ctime +30 -exec rm -rf '{}' ';' + find /data/workspace/ -type d -ctime +90 -exec rm -rf '{}' ';' + find '/root/workspace/*/logs/*' -type d -ctime +30 -exec rm -rf '{}' ';' find: ‘/root/workspace/*/logs/*’: No such file or directory + find /root/workspace/ -type d -ctime +90 -exec rm -rf '{}' ';' find: ‘/root/workspace/’: No such file or directory + killall --older-than 240h python python: no process found + killall --older-than 10h jython jython: no process found + echo -d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/ -d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/ + echo + '[' -z '-d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/' ']' + '[' -z '-d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/' ']' + rerun_param='-d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/' + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.9652.ini -c conf/collections/collections_failover.conf -p GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -m rest -d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/' > Configure project : Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/testexec.9652.ini -c conf/collections/collections_failover.conf -p GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -m rest -d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/ Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/testexec.9652.ini -c conf/collections/collections_failover.conf -p GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -m rest -d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/ Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/testexec.9652.ini -c conf/collections/collections_failover.conf -p GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -m rest -d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/ Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/testexec.9652.ini -c conf/collections/collections_failover.conf -p GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -m rest -d failed=http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523/ > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_failover.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Downloading http://qa.sc.couchbase.com/job/test_suite_executor-TAF/42523//testReport/api/xml?pretty=true to logs/test_suite_executor-TAF_42523__testresult.xml Loading result data from logs/test_suite_executor-TAF_42523__testresult.xml -- logs/test_suite_executor-TAF_42523__testresult.xml -- Global Test input params: {'GROUP': 'P0_step_wise_failover_and_recovery', 'cluster_name': 'testexec.9652', 'conf_file': 'conf/collections/collections_failover.conf', 'ini': '/tmp/testexec.9652.ini', 'num_nodes': 5, 'rerun': 'False', 'spec': 'collections_failover', 'upgrade_version': '7.0.0-2588'} Only cases in GROUPs 'P0_step_wise_failover_and_recovery' will be executed Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_1 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,data_load_stage=before,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=full,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'before', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'full', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_1', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 1, u'infra_log_level': u'critical'} WARNING: Failed to import concurrent.futures module test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... ok ---------------------------------------------------------------------- Ran 1 test in 1390.094s OK During the test, Remote Connections: 140, Disconnections: 135 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_1 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_2 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,data_load_stage=after,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=full,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'after', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'full', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_2', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 2, u'infra_log_level': u'critical'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2020-07-12 20:07:33,973 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.108.211 - Core dump seen: 1 172.23.108.211 : Stack Trace of first crash: b24a90da-2772-42df-011a3bb3-bb2e0055.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fecd37455f7 in raise () from /lib64/libc.so.6 #0 0x00007fecd37455f7 in raise () from /lib64/libc.so.6 #1 0x00007fecd3746ce8 in abort () from /lib64/libc.so.6 #2 0x00007fecd4297195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007fecd4294f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007fecd4294fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007fecd4295213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007fecd79c1b14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007fecd7b99aa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007fecd7b60c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007fecd7717acb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007fecd77174a1 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007fecd77177c4 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007fecd7718439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007fecd772c500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #15 0x00007fecd7b6d868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007fecd7b6e79a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007fecd7b6ee35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007fecd7a55982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007fecd7a5914d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #20 0x00007fecd7ab2d7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #21 0x00007fecd7ab3f49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #22 0x00007fecd7ab6ef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007fecd7aad69f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #24 0x00007fecd649b777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #25 0x00007fecd3ad8dc5 in start_thread () from /lib64/libpthread.so.0 #26 0x00007fecd380621d in clone () from /lib64/libc.so.6 2020-07-12 20:12:52,687 | test | ERROR | MainThread | [rest_client:_http_request:746] POST http://172.23.108.211:8091/controller/startLogsCollection body: nodes=ns_1%40172.23.108.211 headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown {"_":"Logs collection task is already started"} auth: Administrator:password 2020-07-12 20:13:02,700 | test | ERROR | MainThread | [basetestcase:fetch_cb_collect_logs:570] API perform_cb_collect returned False 2020-07-12 20:13:02,884 | test | ERROR | MainThread | [rest_client:_http_request:746] POST http://172.23.121.216:8091/controller/startLogsCollection body: nodes=ns_1%40172.23.121.216 headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown {"_":"Logs collection task is already started"} auth: Administrator:password 2020-07-12 20:13:12,908 | test | ERROR | MainThread | [basetestcase:fetch_cb_collect_logs:570] API perform_cb_collect returned False During the test, Remote Connections: 268, Disconnections: 258 SDK Connections: 60, Disconnections: 60 ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 36, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 33, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 371, in tearDown self.assertEqual(len(server_with_crashes), 0, AssertionError: Test failed, Coredump found on servers ['172.23.108.211'] ---------------------------------------------------------------------- Ran 1 test in 2118.592s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 1 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_2 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_3 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,data_load_stage=after,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=delta,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'after', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'delta', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_3', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 3, u'infra_log_level': u'critical'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2020-07-12 20:42:15,584 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.108.210 - Core dump seen: 4 172.23.108.210 : Stack Trace of first crash: d7117135-873d-4092-316743a5-4dddcab1.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fe58cb0e5f7 in raise () from /lib64/libc.so.6 #0 0x00007fe58cb0e5f7 in raise () from /lib64/libc.so.6 #1 0x00007fe58cb0fce8 in abort () from /lib64/libc.so.6 #2 0x00007fe58d660195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007fe58d65df86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007fe58d65dfd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007fe58d65e213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007fe590d8ab14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007fe590f62aa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007fe590f29c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007fe590ae0acb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007fe590ae04a1 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007fe590ae1439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007fe590af5500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007fe590f36868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #15 0x00007fe590f3779a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007fe590f37e35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007fe590e1e982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007fe590e2214d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007fe590e7bd7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #20 0x00007fe590e7cf49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #21 0x00007fe590e7fef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #22 0x00007fe590e7669f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007fe58f864777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #24 0x00007fe58cea1dc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fe58cbcf21d in clone () from /lib64/libc.so.6 2020-07-12 20:42:19,318 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.108.211 - Core dump seen: 4 172.23.108.211 : Stack Trace of first crash: 10d0d173-8b35-4913-5202b2aa-6d48faf9.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fa4264685f7 in raise () from /lib64/libc.so.6 #0 0x00007fa4264685f7 in raise () from /lib64/libc.so.6 #1 0x00007fa426469ce8 in abort () from /lib64/libc.so.6 #2 0x00007fa426fba195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007fa426fb7f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007fa426fb7fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007fa426fb8213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007fa42a6e4b14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007fa42a8bcaa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007fa42a883c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007fa42a43aacb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007fa42a43a4a1 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007fa42a43b439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007fa42a44f500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007fa42a890868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #15 0x00007fa42a89179a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007fa42a891e35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007fa42a778982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007fa42a77c14d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007fa42a7d5d7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #20 0x00007fa42a7d6f49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #21 0x00007fa42a7d9ef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #22 0x00007fa42a7d069f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007fa4291be777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #24 0x00007fa4267fbdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007fa42652921d in clone () from /lib64/libc.so.6 ERROR ====================================================================== ERROR: test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 36, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 33, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 371, in tearDown self.assertEqual(len(server_with_crashes), 0, AssertionError: Test failed, Coredump found on servers ['172.23.108.210', '172.23.108.211'] During the test, Remote Connections: 398, Disconnections: 383 SDK Connections: 90, Disconnections: 90 ---------------------------------------------------------------------- Ran 1 test in 2560.223s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 2 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_3 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_4 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,data_load_stage=during,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=delta,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'during', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'delta', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_4', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 4, u'infra_log_level': u'critical'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... ok ---------------------------------------------------------------------- Ran 1 test in 1276.431s OK During the test, Remote Connections: 523, Disconnections: 503 SDK Connections: 120, Disconnections: 120 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 2 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_4 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_5 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,data_load_stage=before,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=delta,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'before', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'delta', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_5', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 5, u'infra_log_level': u'critical'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2020-07-12 21:32:22,471 | test | ERROR | pool-7-thread-7 | [task:call:3386] Failed to load 3 docs from 1000 to 1960 2020-07-12 21:32:22,474 | test | ERROR | pool-7-thread-7 | [task:call:3386] Failed to load 16 docs from 0 to 1000 ok ---------------------------------------------------------------------- Ran 1 test in 1290.702s OK During the test, Remote Connections: 648, Disconnections: 623 SDK Connections: 150, Disconnections: 150 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 2 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_5 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_6 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,data_load_stage=before,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=full,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'before', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'full', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_6', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 6, u'infra_log_level': u'critical'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... ok ---------------------------------------------------------------------- Ran 1 test in 1325.095s OK During the test, Remote Connections: 773, Disconnections: 743 SDK Connections: 180, Disconnections: 180 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 2 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_6 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_7 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,data_load_stage=during,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=delta,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'during', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'delta', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_7', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 7, u'infra_log_level': u'critical'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... ok ---------------------------------------------------------------------- Ran 1 test in 1353.582s OK During the test, Remote Connections: 898, Disconnections: 863 SDK Connections: 210, Disconnections: 210 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 2 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_7 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_8 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,data_load_stage=after,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=delta,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'after', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'delta', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_8', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 8, u'infra_log_level': u'critical'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2020-07-12 22:50:50,111 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.108.210 - Core dump seen: 2 172.23.108.210 : Stack Trace of first crash: 54e932bc-6354-4297-4cf37ebb-0441aa5b.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007ff04a24b5f7 in raise () from /lib64/libc.so.6 #0 0x00007ff04a24b5f7 in raise () from /lib64/libc.so.6 #1 0x00007ff04a24cce8 in abort () from /lib64/libc.so.6 #2 0x00007ff04ad9d195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007ff04ad9af86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007ff04ad9afd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007ff04ad9b213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007ff04e4c7b14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007ff04e69faa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007ff04e666c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007ff04e21dacb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007ff04e21d7c4 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007ff04e21e439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007ff04e232500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007ff04e673868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #15 0x00007ff04e67479a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007ff04e674e35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007ff04e55b982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007ff04e55f14d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007ff04e5b8d7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #20 0x00007ff04e5b9f49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #21 0x00007ff04e5bcef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #22 0x00007ff04e5b369f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007ff04cfa1777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #24 0x00007ff04a5dedc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007ff04a30c21d in clone () from /lib64/libc.so.6 2020-07-12 22:50:52,630 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.108.211 - Core dump seen: 2 172.23.108.211 : Stack Trace of first crash: f523c0c1-d8d4-48d3-f12be58a-506af7da.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007ff9d7a495f7 in raise () from /lib64/libc.so.6 #0 0x00007ff9d7a495f7 in raise () from /lib64/libc.so.6 #1 0x00007ff9d7a4ace8 in abort () from /lib64/libc.so.6 #2 0x00007ff9d859b195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007ff9d8598f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007ff9d8598fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007ff9d8599213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007ff9dbcc5b14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007ff9dbe9daa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007ff9dbe64c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007ff9dba1bacb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007ff9dba1b4a1 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007ff9dba1c439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007ff9dba30500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007ff9dbe71868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #15 0x00007ff9dbe7279a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007ff9dbe72e35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007ff9dbd59982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007ff9dbd5d14d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007ff9dbdb6d7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #20 0x00007ff9dbdb7f49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #21 0x00007ff9dbdbaef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #22 0x00007ff9dbdb169f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007ff9da79f777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #24 0x00007ff9d7ddcdc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007ff9d7b0a21d in clone () from /lib64/libc.so.6 2020-07-12 22:56:10,213 | test | ERROR | MainThread | [rest_client:_http_request:746] POST http://172.23.108.211:8091/controller/startLogsCollection body: nodes=ns_1%40172.23.108.211 headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown {"_":"Logs collection task is already started"} auth: Administrator:password 2020-07-12 22:56:20,223 | test | ERROR | MainThread | [basetestcase:fetch_cb_collect_logs:570] API perform_cb_collect returned False ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 36, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 33, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 371, in tearDown self.assertEqual(len(server_with_crashes), 0, AssertionError: Test failed, Coredump found on servers ['172.23.108.210', '172.23.108.211'] ---------------------------------------------------------------------- Ran 1 test in 2177.270s During the test, Remote Connections: 1027, Disconnections: 987 SDK Connections: 240, Disconnections: 240 FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 3 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_8 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_9 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,data_load_stage=during,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=full,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'during', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'full', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_9', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 9, u'infra_log_level': u'critical'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... ok ---------------------------------------------------------------------- Ran 1 test in 1443.923s During the test, Remote Connections: 1152, Disconnections: 1107 SDK Connections: 270, Disconnections: 270 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 6 , fail 3 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_9 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_10 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,data_load_stage=before,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=delta,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'before', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'delta', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_10', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 10, u'infra_log_level': u'critical'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2020-07-12 23:52:31,463 | test | ERROR | MainThread | [bucket_ready_functions:delete_bucket:1412] Unable to get timings for bucket: global name 'StatsCommon' is not defined ok ---------------------------------------------------------------------- Ran 1 test in 1364.204s OK During the test, Remote Connections: 1279, Disconnections: 1229 SDK Connections: 300, Disconnections: 300 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 7 , fail 3 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_10 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_11 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,data_load_stage=after,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=full,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'after', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'full', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_11', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 11, u'infra_log_level': u'critical'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2020-07-13 00:15:18,940 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.108.210 - Core dump seen: 1 172.23.108.210 : Stack Trace of first crash: a67acdc7-cff0-4ad1-338033aa-c16e0085.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f57b49ae5f7 in raise () from /lib64/libc.so.6 #0 0x00007f57b49ae5f7 in raise () from /lib64/libc.so.6 #1 0x00007f57b49afce8 in abort () from /lib64/libc.so.6 #2 0x00007f57b5500195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007f57b54fdf86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007f57b54fdfd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007f57b54fe213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007f57b8c2ab14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007f57b8e02aa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007f57b8dc9c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007f57b8980acb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007f57b89807c4 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007f57b8981439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007f57b8995500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007f57b8dd6868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #15 0x00007f57b8dd779a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007f57b8dd7e35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007f57b8cbe982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007f57b8cc214d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007f57b8d1bd7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #20 0x00007f57b8d1cf49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #21 0x00007f57b8d1fef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #22 0x00007f57b8d1669f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007f57b7704777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #24 0x00007f57b4d41dc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f57b4a6f21d in clone () from /lib64/libc.so.6 2020-07-13 00:15:21,398 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.108.211 - Core dump seen: 1 172.23.108.211 : Stack Trace of first crash: 84dcb940-ee5c-4811-861f12a0-79651867.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f8623e645f7 in raise () from /lib64/libc.so.6 #0 0x00007f8623e645f7 in raise () from /lib64/libc.so.6 #1 0x00007f8623e65ce8 in abort () from /lib64/libc.so.6 #2 0x00007f86249b6195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007f86249b3f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007f86249b3fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007f86249b4213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007f86280e0b14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007f86282b8aa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007f862827fc32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007f8627e36acb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007f8627e367c4 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007f8627e37439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007f8627e4b500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007f862828c868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #15 0x00007f862828d79a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007f862828de35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007f8628174982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007f862817814d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007f86281d1d7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #20 0x00007f86281d2f49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #21 0x00007f86281d5ef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #22 0x00007f86281cc69f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007f8626bba777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #24 0x00007f86241f7dc5 in start_thread () from /lib64/libpthread.so.0 #25 0x00007f8623f2521d in clone () from /lib64/libc.so.6 2020-07-13 00:15:24,345 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.122.63 - Core dump seen: 1 172.23.122.63 : Stack Trace of first crash: efed12c8-7429-4158-6bed55ae-ecfc454f.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f0dd00ea337 in raise () from /lib64/libc.so.6 #0 0x00007f0dd00ea337 in raise () from /lib64/libc.so.6 #1 0x00007f0dd00eba28 in abort () from /lib64/libc.so.6 #2 0x00007f0dd0c48195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007f0dd0c45f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007f0dd0c45fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007f0dd0c46213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007f0dd4372b14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007f0dd454aaa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007f0dd4511c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007f0dd40c8acb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007f0dd40c84a1 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007f0dd40c87c4 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007f0dd40c9439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007f0dd40dd500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #15 0x00007f0dd451e868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007f0dd451f79a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007f0dd451fe35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007f0dd4406982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007f0dd440a14d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #20 0x00007f0dd4463d7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #21 0x00007f0dd4464f49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #22 0x00007f0dd4467ef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007f0dd445e69f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #24 0x00007f0dd2e4c777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #25 0x00007f0dd0489e65 in start_thread () from /lib64/libpthread.so.0 #26 0x00007f0dd01b288d in clone () from /lib64/libc.so.6 2020-07-13 00:15:25,812 | test | ERROR | MainThread | [basetestcase:check_coredump_exist:633] Node 172.23.122.64 - Core dump seen: 1 172.23.122.64 : Stack Trace of first crash: 8d8e7aa7-95be-4bc2-85c79abe-d4d39a47.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007feead0ec337 in raise () from /lib64/libc.so.6 #0 0x00007feead0ec337 in raise () from /lib64/libc.so.6 #1 0x00007feead0eda28 in abort () from /lib64/libc.so.6 #2 0x00007feeadc4a195 in __gnu_cxx::__verbose_terminate_handler() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #3 0x000000000054d8e2 in backtrace_terminate_handler() () #4 0x00007feeadc47f86 in __cxxabiv1::__terminate(void (*)()) () from /opt/couchbase/bin/../lib/libstdc++.so.6 #5 0x00007feeadc47fd1 in std::terminate() () from /opt/couchbase/bin/../lib/libstdc++.so.6 #6 0x00007feeadc48213 in __cxa_throw () from /opt/couchbase/bin/../lib/libstdc++.so.6 #7 0x00007feeb1374b14 in cb::ThrowExceptionUnderflowPolicy::underflow(unsigned long&, unsigned long, long) () from /opt/couchbase/bin/../lib/libep.so #8 0x00007feeb154caa1 in Collections::VB::Flush::updateDiskSize(DocKey const&, long) () from /opt/couchbase/bin/../lib/libep.so #9 0x00007feeb1513c32 in saveDocsCallback(_docinfo const*, _docinfo const*, void*) () from /opt/couchbase/bin/../lib/libep.so #10 0x00007feeb10caacb in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #11 0x00007feeb10ca4a1 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #12 0x00007feeb10ca7c4 in modify_node(couchfile_modify_request*, node_pointer*, int, int, couchfile_modify_result*) () from /opt/couchbase/bin/../lib/libcouchstore.so #13 0x00007feeb10cb439 in modify_btree(couchfile_modify_request*, node_pointer*, couchstore_error_t*) () from /opt/couchbase/bin/../lib/libcouchstore.so #14 0x00007feeb10df500 in couchstore_save_documents_and_callback () from /opt/couchbase/bin/../lib/libcouchstore.so #15 0x00007feeb1520868 in CouchKVStore::saveDocs(Vbid, std::vector<_doc*, std::allocator<_doc*> > const&, std::vector<_docinfo*, std::allocator<_docinfo*> >&, kvstats_ctx&) () from /opt/couchbase/bin/../lib/libep.so #16 0x00007feeb152179a in CouchKVStore::commit2couchstore(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #17 0x00007feeb1521e35 in CouchKVStore::commit(VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #18 0x00007feeb1408982 in EPBucket::commit(Vbid, KVStore&, VB::Commit&) () from /opt/couchbase/bin/../lib/libep.so #19 0x00007feeb140c14d in EPBucket::flushVBucket(Vbid) () from /opt/couchbase/bin/../lib/libep.so #20 0x00007feeb1465d7c in Flusher::flushVB() () from /opt/couchbase/bin/../lib/libep.so #21 0x00007feeb1466f49 in Flusher::step(GlobalTask*) () from /opt/couchbase/bin/../lib/libep.so #22 0x00007feeb1469ef3 in GlobalTask::execute() () from /opt/couchbase/bin/../lib/libep.so #23 0x00007feeb146069f in ExecutorThread::run() () from /opt/couchbase/bin/../lib/libep.so #24 0x00007feeafe4e777 in platform_thread_wrap(void*) () from /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0 #25 0x00007feead48be65 in start_thread () from /lib64/libpthread.so.0 #26 0x00007feead1b488d in clone () from /lib64/libc.so.6 2020-07-13 00:20:39,683 | test | ERROR | MainThread | [rest_client:_http_request:746] POST http://172.23.108.211:8091/controller/startLogsCollection body: nodes=ns_1%40172.23.108.211 headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown {"_":"Logs collection task is already started"} auth: Administrator:password 2020-07-13 00:20:49,690 | test | ERROR | MainThread | [basetestcase:fetch_cb_collect_logs:570] API perform_cb_collect returned False 2020-07-13 00:20:49,717 | test | ERROR | MainThread | [rest_client:_http_request:746] POST http://172.23.121.216:8091/controller/startLogsCollection body: nodes=ns_1%40172.23.121.216 headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown {"_":"Logs collection task is already started"} auth: Administrator:password 2020-07-13 00:20:59,727 | test | ERROR | MainThread | [basetestcase:fetch_cb_collect_logs:570] API perform_cb_collect returned False 2020-07-13 00:20:59,753 | test | ERROR | MainThread | [rest_client:_http_request:746] POST http://172.23.122.63:8091/controller/startLogsCollection body: nodes=ns_1%40172.23.122.63 headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown {"_":"Logs collection task is already started"} auth: Administrator:password 2020-07-13 00:21:09,760 | test | ERROR | MainThread | [basetestcase:fetch_cb_collect_logs:570] API perform_cb_collect returned False During the test, Remote Connections: 1406, Disconnections: 1351 SDK Connections: 330, Disconnections: 330 ERROR ====================================================================== ERROR: test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 36, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 33, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 371, in tearDown self.assertEqual(len(server_with_crashes), 0, AssertionError: Test failed, Coredump found on servers ['172.23.108.210', '172.23.108.211', '172.23.122.63', '172.23.122.64'] ---------------------------------------------------------------------- Ran 1 test in 1908.749s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 7 , fail 4 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_11 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_12 ./testrunner -i /tmp/testexec.9652.ini GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,data_load_stage=during,step_count=1,upgrade_version=7.0.0-2588,rerun=False,GROUP=P0_step_wise_failover_and_recovery,bucket_spec=multi_bucket.buckets_for_rebalance_tests,nodes_failover=2,get-cbcollect-info=True,recovery_type=full,log_level=error,nodes_init=5,infra_log_level=critical Test Input params: {u'data_load_stage': u'during', 'cluster_name': 'testexec.9652', u'nodes_failover': u'2', u'get-cbcollect-info': u'True', 'ini': '/tmp/testexec.9652.ini', u'step_count': u'1', u'recovery_type': u'full', u'log_level': u'error', 'conf_file': 'conf/collections/collections_failover.conf', u'upgrade_version': '7.0.0-2588', u'rerun': 'False', 'spec': 'collections_failover', 'num_nodes': 5, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_12', u'GROUP': 'P0_step_wise_failover_and_recovery', u'nodes_init': u'5', u'bucket_spec': u'multi_bucket.buckets_for_rebalance_tests', 'case_number': 12, u'infra_log_level': u'critical'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... ok ---------------------------------------------------------------------- Ran 1 test in 1349.868s OK During the test, Remote Connections: 1531, Disconnections: 1471 SDK Connections: 360, Disconnections: 360 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 8 , fail 4 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/test_12 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 5h 26m 14s 2 actionable tasks: 1 executed, 1 up-to-date + echo workspace is /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0 workspace is /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0 ++ grep 'testsuite errors' ++ cat /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/report-20-Jul-12_19-21-29-bucket_collections.collections_rebalance.CollectionsRebalance.xml ++ awk '{split($3,s1,"=");print s1[2]}' ++ sed 's/"//g' ++ awk '{s+=$1} END {print s}' + fails=4 + echo fails is 4 fails is 4 ++ grep 'testsuite errors' ++ cat /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-12_19-21-29/report-20-Jul-12_19-21-29-bucket_collections.collections_rebalance.CollectionsRebalance.xml ++ awk '{split($6,s1,"=");print s1[2]}' ++ sed 's/"//g' ++ awk '{s+=$1} END {print s}' + total_tests=12 + echo 12 12 + echo Desc1: 7.0.0-2588 - collections step_wise_failover_and_recovery_P0 - centos '(8/12)' Desc1: 7.0.0-2588 - collections step_wise_failover_and_recovery_P0 - centos (8/12) + '[' true == true ']' + guides/gradlew --stacktrace rerun_job -P jython=/opt/jython/bin/jython -P 'args=7.0.0-2588 --executor_jenkins_job --run_params=GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588' > Configure project : Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py 7.0.0-2588 --executor_jenkins_job --run_params=GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py 7.0.0-2588 --executor_jenkins_job --run_params=GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py 7.0.0-2588 --executor_jenkins_job --run_params=GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.0.1-SNAPSHOT/c6ff42a1c4529a1d3645ef0a70df43588ac99b15/couchbase-transactions-1.0.1-SNAPSHOT.jar:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.ini4j/ini4j/0.5.1/8e737d82ecac9ba6100a9bba71e92a381b75efdc/ini4j-0.5.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.guava/guava/22.0/3564ef3803de51fb0530a8377ec6100b33b0d073/guava-22.0.jar:/root/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.4/c51c00206bb913cd8612b24abd9fa98ae89719b1/commons-cli-1.4.jar:/root/.gradle/caches/modules-2/files-2.1/net.sourceforge.argparse4j/argparse4j/0.8.1/2c8241f84acf6c924bd75be0dbd68e8d74fbcd70/argparse4j-0.8.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.5/50e852b87fb272cdfcaf77097bef8f253a7a863c/java-client-3.0.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.code.findbugs/jsr305/1.3.9/40719ea6961c0cb6afaeb6a921eaa1f6afd4cfdf/jsr305-1.3.9.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.0.18/5f65affce1684999e2f4024983835efc3504012e/error_prone_annotations-2.0.18.jar:/root/.gradle/caches/modules-2/files-2.1/com.google.j2objc/j2objc-annotations/1.1/ed28ded51a8b1c6b112568def5f4b455e6809019/j2objc-annotations-1.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.codehaus.mojo/animal-sniffer-annotations/1.14/775b7e22fb10026eed3f86e8dc556dfafe35f2d5/animal-sniffer-annotations-1.14.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.6/786e661fb89e48d3bf45f84b5ba94a97152ba1ec/core-io-2.0.6.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.5.RELEASE/3fe17d2b3c413c38f728049fd334ba9f69b416e1/reactor-core-3.3.5.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py 7.0.0-2588 --executor_jenkins_job --run_params=GROUP=P0_step_wise_failover_and_recovery,rerun=False,upgrade_version=7.0.0-2588 > Task :compileJava UP-TO-DATE > Task :rerun_job reading from centos_collections_step_wise_failover_and_recovery_P0_7.0.0-2588 upserted centos_collections_step_wise_failover_and_recovery_P0_7.0.0-2588 Merging xmls INFO:merge_reports:Merging of report files from ['Old_Report_mergedreport-20-Jul-11_21-32-07-bucket_collections.collections_rebalance.CollectionsRebalance.xml', 'logs/**/*.xml'] INFO:merge_reports:-- Old_Report_mergedreport-20-Jul-11_21-32-07-bucket_collections.collections_rebalance.CollectionsRebalance.xml -- INFO:merge_reports:-- logs/testrunner-20-Jul-12_19-21-29/report-20-Jul-12_19-21-29-bucket_collections.collections_rebalance.CollectionsRebalance.xml -- INFO:merge_reports: Number of TestSuites=1 INFO:merge_reports: TestSuite#1) bucket_collections.collections_rebalance.CollectionsRebalance, Number of Tests=12 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 8 , fail 4 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery INFO:merge_reports:Summary file is at /data/workspace/centos-p0-collections-vset00-00-step_wise_failover_and_recovery_P0/logs/testrunner-20-Jul-13_00-47-49/merged_summary/mergedreport-20-Jul-13_00-47-49-bucket_collections.collections_rebalance.CollectionsRebalance.xml merged xmls No more failed tests. Stopping reruns Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.4/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 16s 2 actionable tasks: 1 executed, 1 up-to-date [description-setter] Description set: 7.0.0-2588 - collections step_wise_failover_and_recovery_P0 - centos (8/12) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Request made to compress build log Sending email to: sumedh.basarkod@couchbase.com balakumaran.gopal@couchbase.com Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Finished: UNSTABLE