Started by remote host 172.23.107.166 [EnvInject] - Loading node environment variables. Building remotely on slv-s72502 (P0 jython_slave PY3P0) in workspace /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Running Prebuild steps [centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1] $ /bin/sh -xe /tmp/jenkins7475320947537651180.sh ++ echo collections-rebalance_sdk_compression_7.0_P1-Mar-24-22:02:58-7.0.0-4774 ++ awk '{split($0,r,"-");print r[1],r[2]}' + desc='collections rebalance_sdk_compression_7.0_P1' + echo Desc: 7.0.0-4774 - collections rebalance_sdk_compression_7.0_P1 - centos Desc: 7.0.0-4774 - collections rebalance_sdk_compression_7.0_P1 - centos + echo newState=available + newState=available Success build forhudson.tasks.Shell@3f4dce2c [description-setter] Description set: 7.0.0-4774 - collections rebalance_sdk_compression_7.0_P1 - centos Success build forhudson.plugins.descriptionsetter.DescriptionSetterBuilder@137472c [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Success build fororg.jenkinsci.plugins.envinject.EnvInjectBuilder@266fa577 Cloning the remote Git repository Using shallow clone Cloning repository git://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1 # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=30 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 6159fcfb950d06465b761510ab89b2d107b4434a (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 6159fcfb950d06465b761510ab89b2d107b4434a > /usr/bin/git rev-list 6159fcfb950d06465b761510ab89b2d107b4434a # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #102732 jenkins-test_suite_executor-TAF-102732 # timeout=10 No emails were triggered. [centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1] $ /bin/sh -xe /tmp/jenkins5166501605500592813.sh + echo Desc: collections-rebalance_sdk_compression_7.0_P1-Mar-24-22:02:58-7.0.0-4774 Desc: collections-rebalance_sdk_compression_7.0_P1-Mar-24-22:02:58-7.0.0-4774 [description-setter] Could not determine description. [centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1] $ /bin/bash /tmp/jenkins895257138202697026.sh Jython 2.7.0 ############################################## Installing dependencies from requirements.txt ############################################## Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... #################### Done #################### [centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1] $ /bin/bash /tmp/jenkins7510756051250500110.sh Cloning into 'guides'... [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'rerun_props_file' [EnvInject] - Variables injected successfully. [centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1] $ /bin/bash /tmp/jenkins6728254541597835900.sh core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 95691 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 202400 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 95691 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 7.0.0-4774 Set ALLOW_HTP to False so test could run. the major release is 7 "172.23.123.124","172.23.123.125","172.23.106.6","172.23.106.8","172.23.105.42","172.23.106.9" Searching for httplib2 Best match: httplib2 0.9.2 Processing httplib2-0.9.2-py2.7.egg httplib2 0.9.2 is already the active version in easy-install.pth Using /usr/lib/python2.7/site-packages/httplib2-0.9.2-py2.7.egg Processing dependencies for httplib2 Finished processing dependencies for httplib2 Requirement already satisfied (use --upgrade to upgrade): requests in /opt/jython/Lib/site-packages Cleaning up... Requirement already satisfied (use --upgrade to upgrade): futures in /opt/jython/Lib/site-packages Cleaning up... Cloning into 'testrunner'... Checking out files: 36% (1068/2930) Checking out files: 37% (1085/2930) Checking out files: 38% (1114/2930) Checking out files: 39% (1143/2930) Checking out files: 40% (1172/2930) Checking out files: 41% (1202/2930) Checking out files: 42% (1231/2930) Checking out files: 43% (1260/2930) Checking out files: 44% (1290/2930) Checking out files: 45% (1319/2930) Checking out files: 46% (1348/2930) Checking out files: 47% (1378/2930) Checking out files: 48% (1407/2930) Checking out files: 49% (1436/2930) Checking out files: 50% (1465/2930) Checking out files: 51% (1495/2930) Checking out files: 52% (1524/2930) Checking out files: 53% (1553/2930) Checking out files: 54% (1583/2930) Checking out files: 55% (1612/2930) Checking out files: 56% (1641/2930) Checking out files: 57% (1671/2930) Checking out files: 58% (1700/2930) Checking out files: 59% (1729/2930) Checking out files: 60% (1758/2930) Checking out files: 61% (1788/2930) Checking out files: 62% (1817/2930) Checking out files: 63% (1846/2930) Checking out files: 64% (1876/2930) Checking out files: 65% (1905/2930) Checking out files: 66% (1934/2930) Checking out files: 67% (1964/2930) Checking out files: 68% (1993/2930) Checking out files: 69% (2022/2930) Checking out files: 70% (2051/2930) Checking out files: 71% (2081/2930) Checking out files: 72% (2110/2930) Checking out files: 73% (2139/2930) Checking out files: 74% (2169/2930) Checking out files: 75% (2198/2930) Checking out files: 76% (2227/2930) Checking out files: 77% (2257/2930) Checking out files: 78% (2286/2930) Checking out files: 79% (2315/2930) Checking out files: 80% (2344/2930) Checking out files: 81% (2374/2930) Checking out files: 82% (2403/2930) Checking out files: 83% (2432/2930) Checking out files: 84% (2462/2930) Checking out files: 85% (2491/2930) Checking out files: 86% (2520/2930) Checking out files: 87% (2550/2930) Checking out files: 88% (2579/2930) Checking out files: 89% (2608/2930) Checking out files: 90% (2637/2930) Checking out files: 91% (2667/2930) Checking out files: 92% (2696/2930) Checking out files: 93% (2725/2930) Checking out files: 94% (2755/2930) Checking out files: 95% (2784/2930) Checking out files: 96% (2813/2930) Checking out files: 97% (2843/2930) Checking out files: 98% (2872/2930) Checking out files: 99% (2901/2930) Checking out files: 100% (2930/2930) Checking out files: 100% (2930/2930), done. [global] username:root password:couchbase [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:dynamic port:8091 [_2] ip:dynamic port:8091 [_3] ip:dynamic port:8091 [_4] ip:dynamic port:8091 [_5] ip:dynamic port:8091 [_6] ip:dynamic port:8091 python3 scripts/populateIni.py -s "172.23.123.124","172.23.123.125","172.23.106.6","172.23.106.8","172.23.105.42","172.23.106.9" -d None -a None -i /tmp/testexec_reformat.5901.ini -p centos -o /tmp/testexec.5901.ini -k {} INFO:root:SSH Connecting to 172.23.123.124 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.8 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.9 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.123.125 with username:root, attempt#1 of 5 INFO:root:SSH Connecting to 172.23.105.42 with username:root, attempt#1 of 5 INFO:root:SSH Connected to 172.23.123.125 as root INFO:root:SSH Connected to 172.23.123.124 as root INFO:root:SSH Connected to 172.23.105.42 as root INFO:root:SSH Connected to 172.23.106.8 as root INFO:root:SSH Connected to 172.23.106.6 as root INFO:root:SSH Connected to 172.23.106.9 as root INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:os_distro: CentOS, os_version: centos 7, is_linux_distro: True INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.125: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.123.124: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.105.42: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.8: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.6: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root INFO:root:command executed successfully with root INFO:root:extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 INFO:root:running command.raw on 172.23.106.9: sh -c 'if [[ "$OSTYPE" == "darwin"* ]]; then sysctl hw.memsize|grep -Eo [0-9]; else grep MemTotal /proc/meminfo|grep -Eo [0-9]; fi' INFO:root:command executed successfully with root in main the ini file is /tmp/testexec_reformat.5901.ini the given server info is "172.23.123.124","172.23.123.125","172.23.106.6","172.23.106.8","172.23.105.42","172.23.106.9" Collecting memory info from 172.23.123.125 Collecting memory info from 172.23.123.124 Collecting memory info from 172.23.105.42 Collecting memory info from 172.23.106.8 Collecting memory info from 172.23.106.6 Collecting memory info from 172.23.106.9 the servers memory info is [('172.23.123.125', 4103196), ('172.23.123.124', 4103196), ('172.23.106.8', 4103360), ('172.23.106.6', 4103360), ('172.23.105.42', 4103364), ('172.23.106.9', 4103364)] [global] username:root password:couchbase [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 [_1] ip:172.23.123.125 port:8091 [_2] ip:172.23.123.124 port:8091 [_3] ip:172.23.106.8 port:8091 [_4] ip:172.23.106.6 port:8091 [_5] ip:172.23.105.42 port:8091 [_6] ip:172.23.106.9 port:8091 extra install is Starting a Gradle Daemon, 50 busy Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/testexec_root.5901.ini iptables -F Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/testexec_root.5901.ini iptables -F Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/testexec_root.5901.ini iptables -F Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/testexec_root.5901.ini iptables -F Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/testexec_root.5901.ini iptables -F > Task :iptables 172.23.123.124 172.23.105.42 172.23.123.125 172.23.106.8 172.23.106.9 172.23.106.6 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 28s 1 actionable task: 1 executed Already on 'master' From https://github.com/couchbase/testrunner * branch master -> FETCH_HEAD Already up-to-date. 7.0.0-4774 2021-03-24 22:04:38,012 - root - WARNING - URL: is not valid, will use version to locate build 2021-03-24 22:04:38,016 - root - INFO - SSH Connecting to 172.23.123.125 with username:root, attempt#1 of 5 2021-03-24 22:04:38,124 - root - INFO - SSH Connected to 172.23.123.125 as root 2021-03-24 22:04:38,390 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:38,703 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:38,708 - root - INFO - SSH Connecting to 172.23.123.124 with username:root, attempt#1 of 5 2021-03-24 22:04:38,815 - root - INFO - SSH Connected to 172.23.123.124 as root 2021-03-24 22:04:39,103 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:39,425 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:39,428 - root - INFO - SSH Connecting to 172.23.106.8 with username:root, attempt#1 of 5 2021-03-24 22:04:39,540 - root - INFO - SSH Connected to 172.23.106.8 as root 2021-03-24 22:04:39,835 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:40,174 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:40,178 - root - INFO - SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-03-24 22:04:40,294 - root - INFO - SSH Connected to 172.23.106.6 as root 2021-03-24 22:04:40,607 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:40,946 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:40,950 - root - INFO - SSH Connecting to 172.23.105.42 with username:root, attempt#1 of 5 2021-03-24 22:04:41,059 - root - INFO - SSH Connected to 172.23.105.42 as root 2021-03-24 22:04:41,337 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:41,662 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:41,666 - root - INFO - SSH Connecting to 172.23.106.9 with username:root, attempt#1 of 5 2021-03-24 22:04:41,783 - root - INFO - SSH Connected to 172.23.106.9 as root 2021-03-24 22:04:42,115 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:42,441 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:42,446 - root - INFO - SSH Connecting to 172.23.123.125 with username:root, attempt#1 of 5 2021-03-24 22:04:42,548 - root - INFO - SSH Connected to 172.23.123.125 as root 2021-03-24 22:04:42,819 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:43,135 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:43,138 - root - INFO - SSH Connecting to 172.23.123.124 with username:root, attempt#1 of 5 2021-03-24 22:04:43,243 - root - INFO - SSH Connected to 172.23.123.124 as root 2021-03-24 22:04:43,527 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:43,880 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:43,885 - root - INFO - SSH Connecting to 172.23.106.8 with username:root, attempt#1 of 5 2021-03-24 22:04:43,997 - root - INFO - SSH Connected to 172.23.106.8 as root 2021-03-24 22:04:44,306 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:44,645 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:44,648 - root - INFO - SSH Connecting to 172.23.106.6 with username:root, attempt#1 of 5 2021-03-24 22:04:44,763 - root - INFO - SSH Connected to 172.23.106.6 as root 2021-03-24 22:04:45,089 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:45,440 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:45,445 - root - INFO - SSH Connecting to 172.23.105.42 with username:root, attempt#1 of 5 2021-03-24 22:04:45,553 - root - INFO - SSH Connected to 172.23.105.42 as root 2021-03-24 22:04:45,842 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:46,173 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:46,177 - root - INFO - SSH Connecting to 172.23.106.9 with username:root, attempt#1 of 5 2021-03-24 22:04:46,298 - root - INFO - SSH Connected to 172.23.106.9 as root 2021-03-24 22:04:46,649 - root - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-03-24 22:04:46,988 - root - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 2021-03-24 22:04:46,990 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:46,996 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm is live 2021-03-24 22:04:46,996 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,000 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm is live 2021-03-24 22:04:47,000 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,002 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm is live 2021-03-24 22:04:47,003 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,005 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm is live 2021-03-24 22:04:47,005 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,008 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm is live 2021-03-24 22:04:47,008 - root - INFO - Trying to check is this url alive: http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,011 - root - INFO - This url http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm is live 2021-03-24 22:04:47,011 - root - INFO - Downloading build binary to 172.23.123.125:/tmp/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm.. 2021-03-24 22:04:47,040 - root - INFO - running command.raw on 172.23.123.125: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-03-24 22:04:47,118 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,119 - root - INFO - running command.raw on 172.23.123.125: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,175 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,233 - root - INFO - Downloading build binary to 172.23.123.124:/tmp/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm.. 2021-03-24 22:04:47,257 - root - INFO - running command.raw on 172.23.123.124: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-03-24 22:04:47,319 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,320 - root - INFO - running command.raw on 172.23.123.124: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,376 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,434 - root - INFO - Downloading build binary to 172.23.106.8:/tmp/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm.. 2021-03-24 22:04:47,461 - root - INFO - running command.raw on 172.23.106.8: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-03-24 22:04:47,526 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,527 - root - INFO - running command.raw on 172.23.106.8: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,585 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,647 - root - INFO - Downloading build binary to 172.23.106.6:/tmp/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm.. 2021-03-24 22:04:47,675 - root - INFO - running command.raw on 172.23.106.6: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-03-24 22:04:47,740 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,741 - root - INFO - running command.raw on 172.23.106.6: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:47,800 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,864 - root - INFO - Downloading build binary to 172.23.105.42:/tmp/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm.. 2021-03-24 22:04:47,891 - root - INFO - running command.raw on 172.23.105.42: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-03-24 22:04:47,951 - root - INFO - command executed successfully with root 2021-03-24 22:04:47,952 - root - INFO - running command.raw on 172.23.105.42: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:48,007 - root - INFO - command executed successfully with root 2021-03-24 22:04:48,066 - root - INFO - Downloading build binary to 172.23.106.9:/tmp/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm.. 2021-03-24 22:04:48,092 - root - INFO - running command.raw on 172.23.106.9: curl -I http://172.23.126.166/builds/latestbuilds/couchbase-server/cheshire-cat/4774/couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2>&1 | grep Content-Length 2021-03-24 22:04:48,155 - root - INFO - command executed successfully with root 2021-03-24 22:04:48,155 - root - INFO - running command.raw on 172.23.106.9: cd /tmp/ && wc -c couchbase-server-enterprise-7.0.0-4774-centos7.x86_64.rpm 2021-03-24 22:04:48,217 - root - INFO - command executed successfully with root 2021-03-24 22:04:52,903 - root - INFO - Done with uninstall on 172.23.123.124. 2021-03-24 22:04:53,116 - root - INFO - Done with uninstall on 172.23.105.42. 2021-03-24 22:04:53,438 - root - INFO - Done with uninstall on 172.23.106.8. 2021-03-24 22:04:53,605 - root - INFO - Done with uninstall on 172.23.106.6. 2021-03-24 22:04:53,634 - root - INFO - Done with uninstall on 172.23.106.9. 2021-03-24 22:04:53,921 - root - INFO - Done with uninstall on 172.23.123.125. 2021-03-24 22:06:19,668 - root - INFO - Done with install on 172.23.106.9. 2021-03-24 22:06:19,904 - root - INFO - Done with install on 172.23.123.124. 2021-03-24 22:06:19,998 - root - INFO - Done with install on 172.23.123.125. 2021-03-24 22:06:21,969 - root - INFO - Done with install on 172.23.106.6. 2021-03-24 22:06:22,463 - root - INFO - Done with install on 172.23.106.8. 2021-03-24 22:06:26,439 - root - INFO - Done with install on 172.23.105.42. 2021-03-24 22:07:19,729 - root - INFO - running command.raw on 172.23.106.9: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.9 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:19,965 - root - INFO - running command.raw on 172.23.123.124: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.124 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,043 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,044 - root - INFO - running command.raw on 172.23.106.9: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.9 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,054 - root - INFO - running command.raw on 172.23.123.125: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.125 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,257 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,258 - root - INFO - running command.raw on 172.23.123.124: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.124 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,281 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,282 - root - INFO - running command.raw on 172.23.106.9: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.9 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,340 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,340 - root - INFO - running command.raw on 172.23.123.125: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.125 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,478 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,478 - root - INFO - running command.raw on 172.23.123.124: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.124 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,503 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,503 - root - WARNING - Unable to init node 172.23.106.9 2021-03-24 22:07:20,553 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,554 - root - INFO - running command.raw on 172.23.123.125: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.123.125 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:20,741 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,744 - root - WARNING - Unable to init node 172.23.123.124 2021-03-24 22:07:20,772 - root - INFO - command executed successfully with root 2021-03-24 22:07:20,772 - root - WARNING - Unable to init node 172.23.123.125 2021-03-24 22:07:21,517 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.9 2021-03-24 22:07:21,517 - root - INFO - pools/default params : memoryQuota=2147 2021-03-24 22:07:21,523 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-03-24 22:07:21,523 - root - INFO - /node/controller/setupServices params on 172.23.106.9: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-03-24 22:07:21,555 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-03-24 22:07:21,556 - root - INFO - settings/web params on 172.23.106.9:8091:port=8091&username=Administrator&password=password 2021-03-24 22:07:21,617 - root - INFO - --> status:True 2021-03-24 22:07:21,617 - root - INFO - Done with init on 172.23.106.9. 2021-03-24 22:07:21,636 - root - INFO - Done with cleanup on 172.23.106.9. 2021-03-24 22:07:21,758 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.123.124 2021-03-24 22:07:21,758 - root - INFO - pools/default params : memoryQuota=2147 2021-03-24 22:07:21,764 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-03-24 22:07:21,765 - root - INFO - /node/controller/setupServices params on 172.23.123.124: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-03-24 22:07:21,785 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.123.125 2021-03-24 22:07:21,786 - root - INFO - pools/default params : memoryQuota=2147 2021-03-24 22:07:21,791 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-03-24 22:07:21,791 - root - INFO - /node/controller/setupServices params on 172.23.123.125: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-03-24 22:07:21,796 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-03-24 22:07:21,796 - root - INFO - settings/web params on 172.23.123.124:8091:port=8091&username=Administrator&password=password 2021-03-24 22:07:21,821 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-03-24 22:07:21,821 - root - INFO - settings/web params on 172.23.123.125:8091:port=8091&username=Administrator&password=password 2021-03-24 22:07:21,851 - root - INFO - --> status:True 2021-03-24 22:07:21,851 - root - INFO - Done with init on 172.23.123.124. 2021-03-24 22:07:21,873 - root - INFO - Done with cleanup on 172.23.123.124. 2021-03-24 22:07:21,881 - root - INFO - --> status:True 2021-03-24 22:07:21,882 - root - INFO - Done with init on 172.23.123.125. 2021-03-24 22:07:21,904 - root - INFO - Done with cleanup on 172.23.123.125. 2021-03-24 22:07:22,029 - root - INFO - running command.raw on 172.23.106.6: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.6 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:22,329 - root - INFO - command executed successfully with root 2021-03-24 22:07:22,329 - root - INFO - running command.raw on 172.23.106.6: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.6 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:22,523 - root - INFO - running command.raw on 172.23.106.8: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.8 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:22,596 - root - INFO - command executed successfully with root 2021-03-24 22:07:22,596 - root - INFO - running command.raw on 172.23.106.6: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.6 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:22,826 - root - INFO - command executed successfully with root 2021-03-24 22:07:22,827 - root - INFO - running command.raw on 172.23.106.8: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.8 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:22,853 - root - INFO - command executed successfully with root 2021-03-24 22:07:22,853 - root - WARNING - Unable to init node 172.23.106.6 2021-03-24 22:07:23,039 - root - INFO - command executed successfully with root 2021-03-24 22:07:23,040 - root - INFO - running command.raw on 172.23.106.8: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.106.8 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:23,288 - root - INFO - command executed successfully with root 2021-03-24 22:07:23,288 - root - WARNING - Unable to init node 172.23.106.8 2021-03-24 22:07:23,866 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.6 2021-03-24 22:07:23,866 - root - INFO - pools/default params : memoryQuota=2147 2021-03-24 22:07:23,872 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-03-24 22:07:23,872 - root - INFO - /node/controller/setupServices params on 172.23.106.6: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-03-24 22:07:23,902 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-03-24 22:07:23,903 - root - INFO - settings/web params on 172.23.106.6:8091:port=8091&username=Administrator&password=password 2021-03-24 22:07:23,966 - root - INFO - --> status:True 2021-03-24 22:07:23,966 - root - INFO - Done with init on 172.23.106.6. 2021-03-24 22:07:23,990 - root - INFO - Done with cleanup on 172.23.106.6. 2021-03-24 22:07:24,302 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.106.8 2021-03-24 22:07:24,303 - root - INFO - pools/default params : memoryQuota=2147 2021-03-24 22:07:24,308 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-03-24 22:07:24,309 - root - INFO - /node/controller/setupServices params on 172.23.106.8: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-03-24 22:07:24,339 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-03-24 22:07:24,340 - root - INFO - settings/web params on 172.23.106.8:8091:port=8091&username=Administrator&password=password 2021-03-24 22:07:24,399 - root - INFO - --> status:True 2021-03-24 22:07:24,400 - root - INFO - Done with init on 172.23.106.8. 2021-03-24 22:07:24,421 - root - INFO - Done with cleanup on 172.23.106.8. 2021-03-24 22:07:26,499 - root - INFO - running command.raw on 172.23.105.42: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.42 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:26,787 - root - INFO - command executed successfully with root 2021-03-24 22:07:26,787 - root - INFO - running command.raw on 172.23.105.42: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.42 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:27,040 - root - INFO - command executed successfully with root 2021-03-24 22:07:27,042 - root - INFO - running command.raw on 172.23.105.42: /opt/couchbase/bin/couchbase-cli node-init -c 172.23.105.42 -u Administrator -p password > /dev/null && echo 1 || echo 0; 2021-03-24 22:07:27,250 - root - INFO - command executed successfully with root 2021-03-24 22:07:27,251 - root - WARNING - Unable to init node 172.23.105.42 2021-03-24 22:07:28,266 - root - INFO - Setting KV memory quota as 2147 MB on 172.23.105.42 2021-03-24 22:07:28,267 - root - INFO - pools/default params : memoryQuota=2147 2021-03-24 22:07:28,273 - root - INFO - --> init_node_services(Administrator,password,None,8091,['kv']) 2021-03-24 22:07:28,273 - root - INFO - /node/controller/setupServices params on 172.23.105.42: 8091:hostname=None&user=Administrator&password=password&services=kv 2021-03-24 22:07:28,304 - root - INFO - --> in init_cluster...Administrator,password,8091 2021-03-24 22:07:28,305 - root - INFO - settings/web params on 172.23.105.42:8091:port=8091&username=Administrator&password=password 2021-03-24 22:07:28,370 - root - INFO - --> status:True 2021-03-24 22:07:28,370 - root - INFO - Done with init on 172.23.105.42. 2021-03-24 22:07:28,390 - root - INFO - Done with cleanup on 172.23.105.42. 2021-03-24 22:07:28,435 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-24 22:07:28,443 - root - INFO - node:172.23.123.125:8091 version:7.0.0-4774-enterprise aFamily:inet services:['kv'] 2021-03-24 22:07:28,450 - root - INFO - node:172.23.123.124:8091 version:7.0.0-4774-enterprise aFamily:inet services:['kv'] 2021-03-24 22:07:28,456 - root - INFO - node:172.23.106.8:8091 version:7.0.0-4774-enterprise aFamily:inet services:['kv'] 2021-03-24 22:07:28,463 - root - INFO - node:172.23.106.6:8091 version:7.0.0-4774-enterprise aFamily:inet services:['kv'] 2021-03-24 22:07:28,471 - root - INFO - node:172.23.105.42:8091 version:7.0.0-4774-enterprise aFamily:inet services:['kv'] 2021-03-24 22:07:28,476 - root - INFO - node:172.23.106.9:8091 version:7.0.0-4774-enterprise aFamily:inet services:['kv'] 2021-03-24 22:07:28,477 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-24 22:07:28,477 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-24 22:07:28,477 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-24 22:07:28,477 - root - INFO - INSTALL COMPLETED ON: 172.23.123.125 2021-03-24 22:07:28,478 - root - INFO - INSTALL COMPLETED ON: 172.23.123.124 2021-03-24 22:07:28,478 - root - INFO - INSTALL COMPLETED ON: 172.23.106.8 2021-03-24 22:07:28,479 - root - INFO - INSTALL COMPLETED ON: 172.23.106.6 2021-03-24 22:07:28,479 - root - INFO - INSTALL COMPLETED ON: 172.23.105.42 2021-03-24 22:07:28,479 - root - INFO - INSTALL COMPLETED ON: 172.23.106.9 2021-03-24 22:07:28,479 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-03-24 22:07:28,480 - root - INFO - TOTAL INSTALL TIME = 170 seconds Need to set ALLOW_HTP back to True to do git pull branch Switched to a new branch 'master' Branch master set up to track remote branch master from origin. From git://github.com/couchbaselabs/TAF * branch master -> FETCH_HEAD Already up-to-date. Set ALLOW_HTP to False so test could run. HEAD is now at 6159fcf CBQE-6041: Add wipe nodes logic to hard-unsafe failover find: ‘/root/workspace/*/logs/*’: No such file or directory find: ‘/root/workspace/’: No such file or directory python: no process found jython: no process found Timeout: 1200 minutes > Configure project : Using Transaction_client :: 1.1.4 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/testexec.5901.ini -c conf/collections/collections_rebalance.conf -p GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/testexec.5901.ini -c conf/collections/collections_rebalance.conf -p GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/testexec.5901.ini -c conf/collections/collections_rebalance.conf -p GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/testexec.5901.ini -c conf/collections/collections_rebalance.conf -p GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/testexec.5901.ini -c conf/collections/collections_rebalance.conf -p GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -m rest > Task :compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: conf/collections/collections_rebalance.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'GROUP': 'sdk_compression', 'cluster_name': 'testexec.5901', 'conf_file': 'conf/collections/collections_rebalance.conf', 'ini': '/tmp/testexec.5901.ini', 'num_nodes': 6, 'rerun': 'False', 'spec': 'collections_rebalance', 'upgrade_version': '7.0.0-4774'} Only cases in GROUPs 'sdk_compression' will be executed Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_rebalance_out' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=P0_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=P1_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=P1_step_wise_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=bucket_compaction' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,warmup=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=node_warmup' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,warmup=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=node_warmup' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,warmup=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=node_warmup' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,warmup=1,compaction=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=node_warmup' skipped, GROUP not satisfied Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.5901.ini GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec.5901', 'ini': '/tmp/testexec.5901.ini', 'conf_file': 'conf/collections/collections_rebalance.conf', 'upgrade_version': '7.0.0-4774', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_1', 'nodes_init': '3', 'GROUP': 'sdk_compression', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 1, 'sdk_compression': 'True'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-03-24 22:07:56,957 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-24 22:07:57,805 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-24 22:08:00,016 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-24 22:10:33,539 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-24 22:10:50,609 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.123.125, 172.23.123.124, 172.23.106.8, 172.23.106.6, 172.23.105.42, 172.23.106.9 initialized 2021-03-24 22:10:50,647 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-24 22:10:50,693 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-24 22:10:57,329 | test | INFO | pool-1-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 4.10615923886 | Cluster node | | 172.23.123.124 | None | | | <--- IN --- | | 172.23.106.8 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-24 22:10:57,361 | test | INFO | pool-1-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:11:07,497 | test | INFO | pool-1-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:11:07,538 | test | INFO | pool-1-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.2100000381 sec 2021-03-24 22:11:07,898 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-03-24 22:11:15,275 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2261] Waiting for all collections to be created 2021-03-24 22:11:15,611 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 314572800 | 69920016 | 19008464 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 314572800 | 37254640 | 102 | | default | couchbase | 3 | none | 0 | 0 | 4718592000 | 0 | 18457540 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-24 22:11:15,614 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-24 22:12:05,279 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:12:05,321 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:12:23,441 | test | INFO | pool-1-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***.............................*.............................**..........................................**X | 1132.734 | +-------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:12:23,467 | test | INFO | pool-1-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***...................*...................***..............................................................*X | 14802.701 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 22:12:24,871 | test | INFO | pool-1-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***.......*.......**............................***..........................................................X | 2331.083 | +-------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:12:25,033 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.123.124 | kv | 62.3084780388 | 4201672704 | 3273584640 | 43253760 / 3758092288 | 33041 / 65695 | 7.0.0-4774-enterprise | | 172.23.106.8 | kv | 88.4199691199 | 4201840640 | 3152191488 | 0 / 3758092288 | 103467 / 204841 | 7.0.0-4774-enterprise | | 172.23.123.125 | kv | 61.5679264556 | 4201672704 | 3204112384 | 91488256 / 3758092288 | 32941 / 66034 | 7.0.0-4774-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-24 22:12:32,217 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:13:03,823 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 314572800 | 106604680 | 108729454 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 314572800 | 77965400 | 102 | | default | couchbase | 3 | none | 0 | 500000 | 4718592000 | 455320888 | 381511918 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-24 22:13:03,828 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_in ========= 2021-03-24 22:13:07,630 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:822] Doing collection data load during rebalance_in 2021-03-24 22:13:07,631 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:303] Starting rebalance operation of type : rebalance_in 2021-03-24 22:13:07,704 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:13:07,720 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:13:15,101 | test | INFO | pool-1-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.123.124 | kv | 7.0.0-4774-enterprise | 12.4526395554 | Cluster node | | 172.23.106.8 | kv | 7.0.0-4774-enterprise | 11.6514690983 | Cluster node | | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 21.8512898331 | Cluster node | | 172.23.106.6 | None | | | <--- IN --- | | 172.23.105.42 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-24 22:13:15,236 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:13:20,397 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 3.06 2021-03-24 22:13:26,931 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 10.1 2021-03-24 22:13:30,105 | test | INFO | pool-1-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **............................****............................................*............................**X | 1730.600 | +-------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:13:30,237 | test | INFO | pool-1-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...................**.......................................**..........................................**X | 23817.500 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 22:13:31,710 | test | INFO | pool-1-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****.................................**...................................................................**X | 1805.500 | | 1352.300 | .........................................................................X | 1352.300 | +----------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:13:32,323 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 15.68 2021-03-24 22:13:37,417 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 22.99 2021-03-24 22:13:42,526 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 31.62 2021-03-24 22:13:47,631 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 36.84 2021-03-24 22:13:52,723 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 46.13 2021-03-24 22:13:57,825 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 54.57 2021-03-24 22:14:02,898 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 63.68 2021-03-24 22:14:08,009 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 68.87 2021-03-24 22:14:13,099 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 77.93 2021-03-24 22:14:18,190 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 85.11 2021-03-24 22:14:23,292 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 91.33 2021-03-24 22:14:28,377 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: running, progress: 98.17 2021-03-24 22:14:38,499 | test | INFO | pool-1-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:14:38,548 | test | INFO | pool-1-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 83.4590001106 sec 2021-03-24 22:14:47,684 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:15:37,257 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-03-24 22:15:37,259 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-03-24 22:15:37,996 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-03-24 22:15:37,997 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-03-24 22:15:38,786 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: default, Active Resident ratio(DGM): 100% 2021-03-24 22:15:38,788 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: default, Replica Resident ratio(DGM): 100% 2021-03-24 22:15:51,417 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:13:18, Test's start timestamp is 2021-03-24 22:07:56.950000 2021-03-24 22:15:51,417 | test | INFO | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000001.txt 2021-03-24 22:15:51,418 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:791] 172.23.123.124: Found ' ERROR ' logs - ['2021-03-24T22:13:18.648316-07:00 ERROR 1578: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n'] 2021-03-24 22:15:54,575 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1087] Running cbcollect on node ns_1@172.23.105.42,ns_1@172.23.106.6,ns_1@172.23.106.8,ns_1@172.23.123.124,ns_1@172.23.123.125 2021-03-24 22:16:04,634 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1090] ns_1@172.23.105.42,ns_1@172.23.106.6,ns_1@172.23.106.8,ns_1@172.23.123.124,ns_1@172.23.123.125 - cbcollect status: True 2021-03-24 22:16:04,635 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1095] Polling active_tasks to check cbcollect status 2021-03-24 22:19:15,607 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.105.42: Copying cbcollect ZIP file to Client 2021-03-24 22:19:16,960 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.105.42 node cb collect zip copied on client : True 2021-03-24 22:19:16,961 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.6: Copying cbcollect ZIP file to Client 2021-03-24 22:19:18,948 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.6 node cb collect zip copied on client : True 2021-03-24 22:19:18,950 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.8: Copying cbcollect ZIP file to Client 2021-03-24 22:19:20,454 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.8 node cb collect zip copied on client : True 2021-03-24 22:19:20,457 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.124: Copying cbcollect ZIP file to Client 2021-03-24 22:19:22,049 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.124 node cb collect zip copied on client : True 2021-03-24 22:19:22,051 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.125: Copying cbcollect ZIP file to Client 2021-03-24 22:19:23,908 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.125 node cb collect zip copied on client : True 2021-03-24 22:19:40,397 | test | INFO | MainThread | [rest_client:monitorRebalance:1461] Rebalance done. Taken 11.1720001698 seconds to complete 2021-03-24 22:19:40,423 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ERROR ====================================================================== ERROR: test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 87, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 104, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 447, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 713.408s During the test, FAILED (errors=1) Remote Connections: 173, Disconnections: 173 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 1 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_1 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.5901.ini GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec.5901', 'ini': '/tmp/testexec.5901.ini', 'conf_file': 'conf/collections/collections_rebalance.conf', 'upgrade_version': '7.0.0-4774', 'skip_validations': 'False', 'spec': 'collections_rebalance', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_2', 'nodes_init': '5', 'GROUP': 'sdk_compression', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 2, 'sdk_compression': 'True'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-03-24 22:19:49,555 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-03-24 22:19:50,079 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-03-24 22:19:50,637 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-24 22:22:23,986 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-24 22:22:40,201 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.123.125, 172.23.123.124, 172.23.106.8, 172.23.106.6, 172.23.105.42, 172.23.106.9 initialized 2021-03-24 22:22:40,230 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_rebalance_out ========= 2021-03-24 22:22:40,279 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #2 test_data_load_collections_with_rebalance_out ========= 2021-03-24 22:23:03,111 | test | INFO | pool-3-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 4.25851703407 | Cluster node | | 172.23.123.124 | None | | | <--- IN --- | | 172.23.106.8 | None | | | <--- IN --- | | 172.23.106.6 | None | | | <--- IN --- | | 172.23.105.42 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-24 22:23:03,141 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:23:13,233 | test | INFO | pool-3-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:23:13,267 | test | INFO | pool-3-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.1559998989 sec 2021-03-24 22:23:13,543 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-03-24 22:23:21,834 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2261] Waiting for all collections to be created 2021-03-24 22:23:22,161 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 524288000 | 28677728 | 0 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 524288000 | 17387456 | 0 | | default | couchbase | 3 | none | 0 | 0 | 7864320000 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-24 22:23:22,163 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-24 22:24:34,811 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:24:34,828 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:24:52,516 | test | INFO | pool-3-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.*.....................................*...............**..........................*.....................X | 20696.100 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 22:24:52,671 | test | INFO | pool-3-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..*...............................................*.....................**................*..............X | 1898.500 | +-------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:24:52,747 | test | INFO | pool-3-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....*........................................*.............**............................*..............X | 756.000 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-03-24 22:24:52,858 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.123.124 | kv | 53.3350305499 | 4201672704 | 3297947648 | 43253760 / 3758092288 | 32980 / 99727 | 7.0.0-4774-enterprise | | 172.23.106.6 | kv | 74.3989769821 | 4201840640 | 3159654400 | 0 / 3758092288 | 89281 / 269673 | 7.0.0-4774-enterprise | | 172.23.106.8 | kv | 72.8391895358 | 4201840640 | 3208749056 | 0 / 3758092288 | 67099 / 201608 | 7.0.0-4774-enterprise | | 172.23.123.125 | kv | 52.9232333503 | 4201672704 | 3268567040 | 91488256 / 3758092288 | 33168 / 99322 | 7.0.0-4774-enterprise | | 172.23.105.42 | kv | 39.0605054889 | 4201844736 | 3273043968 | 0 / 3758092288 | 36479 / 107116 | 7.0.0-4774-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-24 22:25:14,155 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:26:02,760 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 524288000 | 149725992 | 157476582 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 524288000 | 113298008 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 7864320000 | 616575832 | 470650840 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-24 22:26:02,763 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_rebalance_out ========= 2021-03-24 22:26:09,013 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:822] Doing collection data load during rebalance_out 2021-03-24 22:26:09,015 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:303] Starting rebalance operation of type : rebalance_out 2021-03-24 22:26:09,101 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:26:09,118 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:26:09,266 | test | INFO | pool-3-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.123.124 | kv | 7.0.0-4774-enterprise | 6.60592255125 | Cluster node | | 172.23.106.6 | kv | 7.0.0-4774-enterprise | 9.03324029434 | --- OUT ---> | | 172.23.106.8 | kv | 7.0.0-4774-enterprise | 8.54614412137 | Cluster node | | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 8.00501882058 | Cluster node | | 172.23.105.42 | kv | 7.0.0-4774-enterprise | 8.08823529412 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-24 22:26:09,315 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:26:14,598 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 2.01 2021-03-24 22:26:19,780 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 5.2 2021-03-24 22:26:20,463 | test | INFO | pool-3-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *............*........................**................................................................X | 8901.952 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:26:22,056 | test | INFO | pool-3-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****...........*.........................................................................................X | 406.700 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-03-24 22:26:23,473 | test | INFO | pool-3-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.........................**.........................*.........................*.........................X | 1782.000 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:26:25,273 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 6.37 2021-03-24 22:26:30,414 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 9.07 2021-03-24 22:26:35,522 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 9.77 2021-03-24 22:26:40,618 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 9.99 2021-03-24 22:26:45,711 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 13.94 2021-03-24 22:26:50,789 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 20.26 2021-03-24 22:26:55,861 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 30.68 2021-03-24 22:27:00,931 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 22:27:06,000 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 22:27:11,111 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 38.96 2021-03-24 22:27:16,292 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 50.82 2021-03-24 22:27:21,374 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.99 2021-03-24 22:27:26,444 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 22:27:31,523 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 22:27:36,678 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 68.86 2021-03-24 22:27:41,796 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 80.07 2021-03-24 22:27:46,874 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 88.88 2021-03-24 22:27:51,956 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 97.32 2021-03-24 22:27:57,042 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-03-24 22:28:02,109 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-03-24 22:28:12,362 | test | INFO | pool-3-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:28:12,575 | test | INFO | pool-3-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 123.307999849 sec 2021-03-24 22:28:19,270 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:28:55,147 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-03-24 22:28:55,148 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-03-24 22:28:56,105 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-03-24 22:28:56,105 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-03-24 22:28:57,013 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: default, Active Resident ratio(DGM): 100% 2021-03-24 22:28:57,015 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: default, Replica Resident ratio(DGM): 100% 2021-03-24 22:29:12,026 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:13:18, Test's start timestamp is 2021-03-24 22:19:49.555000 2021-03-24 22:29:15,513 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:28:07, Test's start timestamp is 2021-03-24 22:19:49.555000 2021-03-24 22:29:15,515 | test | INFO | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000004.txt 2021-03-24 22:29:15,516 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:791] 172.23.106.6: Found ' ERROR ' logs - ['2021-03-24T22:26:55.848360-07:00 ERROR 76: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:28:07.341745-07:00 ERROR 1101: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:28:07.380312-07:00 ERROR 1389: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:28:07.767034-07:00 ERROR 1189: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n'] 2021-03-24 22:29:17,128 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1087] Running cbcollect on node ns_1@172.23.106.8,ns_1@172.23.123.124,ns_1@172.23.123.125 2021-03-24 22:29:27,180 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1090] ns_1@172.23.106.8,ns_1@172.23.123.124,ns_1@172.23.123.125 - cbcollect status: True 2021-03-24 22:29:27,180 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1095] Polling active_tasks to check cbcollect status 2021-03-24 22:32:18,562 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.8: Copying cbcollect ZIP file to Client 2021-03-24 22:32:20,566 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.8 node cb collect zip copied on client : True 2021-03-24 22:32:20,569 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.124: Copying cbcollect ZIP file to Client 2021-03-24 22:32:22,184 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.124 node cb collect zip copied on client : True 2021-03-24 22:32:22,184 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.125: Copying cbcollect ZIP file to Client 2021-03-24 22:32:24,302 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.125 node cb collect zip copied on client : True 2021-03-24 22:32:40,457 | test | INFO | MainThread | [rest_client:monitorRebalance:1461] Rebalance done. Taken 11.1870000362 seconds to complete 2021-03-24 22:32:40,489 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ERROR ====================================================================== ERROR: test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 87, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 104, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 447, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 779.939s FAILED (errors=1) During the test, Remote Connections: 173, Disconnections: 173 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 2 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_2 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.5901.ini GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec.5901', 'ini': '/tmp/testexec.5901.ini', 'conf_file': 'conf/collections/collections_rebalance.conf', 'upgrade_version': '7.0.0-4774', 'skip_validations': 'False', 'spec': 'collections_rebalance', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_3', 'nodes_init': '4', 'GROUP': 'sdk_compression', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 3, 'sdk_compression': 'True'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-03-24 22:32:49,186 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-03-24 22:32:49,736 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-03-24 22:32:50,367 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-24 22:35:22,819 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-24 22:35:39,289 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.123.125, 172.23.123.124, 172.23.106.8, 172.23.106.6, 172.23.105.42, 172.23.106.9 initialized 2021-03-24 22:35:39,315 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-03-24 22:35:39,358 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-03-24 22:35:47,851 | test | INFO | pool-4-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 1.5295887663 | Cluster node | | 172.23.123.124 | None | | | <--- IN --- | | 172.23.106.8 | None | | | <--- IN --- | | 172.23.106.6 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-03-24 22:35:47,881 | test | INFO | pool-4-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:35:57,967 | test | INFO | pool-4-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:35:57,999 | test | INFO | pool-4-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.1490001678 sec 2021-03-24 22:35:58,250 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-03-24 22:36:06,673 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2261] Waiting for all collections to be created 2021-03-24 22:36:06,933 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 419430400 | 19134784 | 9629762 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 419430400 | 40539048 | 136 | | default | couchbase | 3 | none | 0 | 0 | 6291456000 | 0 | 5947424 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-24 22:36:06,934 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-24 22:37:04,618 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:37:04,631 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:37:18,997 | test | INFO | pool-4-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **......*......................*........................................................................**X | 12681.057 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 22:37:20,196 | test | INFO | pool-4-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...................*........................*.........................................................**X | 2634.711 | | 2375.511 | .........................................................................................X | 2375.511 | +----------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:37:20,408 | test | INFO | pool-4-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **........*..........................*....................................................***..............X | 2409.912 | +-------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:37:20,516 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.123.124 | kv | 77.0796235055 | 4201672704 | 3193495552 | 43253760 / 3758092288 | 58738 / 175706 | 7.0.0-4774-enterprise | | 172.23.106.6 | kv | 52.2738756948 | 4201840640 | 3173265408 | 0 / 3758092288 | 40647 / 120203 | 7.0.0-4774-enterprise | | 172.23.106.8 | kv | 41.3346866278 | 4201840640 | 3219501056 | 0 / 3758092288 | 15651 / 46271 | 7.0.0-4774-enterprise | | 172.23.123.125 | kv | 78.6259541985 | 4201672704 | 3143712768 | 91488256 / 3758092288 | 58809 / 175252 | 7.0.0-4774-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-24 22:37:28,499 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:38:08,467 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 419430400 | 142627216 | 113786514 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 419430400 | 120086400 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 699849152 | 542660589 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-24 22:38:08,470 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_swap_rebalance ========= 2021-03-24 22:38:13,085 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:822] Doing collection data load during swap_rebalance 2021-03-24 22:38:13,085 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:303] Starting rebalance operation of type : swap_rebalance 2021-03-24 22:38:18,971 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:38:18,983 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:38:19,101 | test | INFO | pool-4-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.106.9 | kv | 7.0.0-4774-enterprise | 0.603469952225 | Cluster node | | 172.23.123.124 | kv | 7.0.0-4774-enterprise | 7.28143109095 | Cluster node | | 172.23.106.6 | kv | 7.0.0-4774-enterprise | 10.2304380856 | --- OUT ---> | | 172.23.106.8 | kv | 7.0.0-4774-enterprise | 9.24092409241 | --- OUT ---> | | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 9.05422446406 | Cluster node | | 172.23.105.42 | kv | 7.0.0-4774-enterprise | 0.824381713715 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-03-24 22:38:19,128 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:38:29,793 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 11.5 2021-03-24 22:38:35,092 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 15.02 2021-03-24 22:38:40,316 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.93 2021-03-24 22:38:41,401 | test | INFO | pool-4-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.......***.............................................................................................X | 6301.862 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:38:41,638 | test | INFO | pool-4-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................................................X | 185.800 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-03-24 22:38:45,476 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 23.87 2021-03-24 22:38:50,733 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 32.16 2021-03-24 22:38:55,941 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 22:39:01,173 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 22:39:06,987 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 38.11 2021-03-24 22:39:12,898 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 47.05 2021-03-24 22:39:18,053 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 54.88 2021-03-24 22:39:23,184 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 61.33 2021-03-24 22:39:28,464 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 22:39:33,704 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 22:39:40,121 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 22:39:45,852 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 72.35 2021-03-24 22:39:51,490 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 77.3 2021-03-24 22:39:56,671 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 81.1 2021-03-24 22:40:01,901 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 87.2 2021-03-24 22:40:07,145 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 93.36 2021-03-24 22:40:12,306 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-03-24 22:40:17,503 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-03-24 22:40:23,305 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-03-24 22:40:34,367 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:40:34,599 | test | INFO | pool-4-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 135.498999834 sec 2021-03-24 22:40:47,232 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:41:37,151 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-03-24 22:41:37,151 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-03-24 22:41:37,868 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-03-24 22:41:37,869 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-03-24 22:41:38,625 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: default, Active Resident ratio(DGM): 100% 2021-03-24 22:41:38,627 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: default, Replica Resident ratio(DGM): 100% 2021-03-24 22:41:51,249 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:13:18, Test's start timestamp is 2021-03-24 22:32:49.185000 2021-03-24 22:41:53,697 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:39:22, Test's start timestamp is 2021-03-24 22:32:49.185000 2021-03-24 22:41:53,698 | test | INFO | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000009.txt 2021-03-24 22:41:53,700 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:791] 172.23.106.8: Found ' ERROR ' logs - ['2021-03-24T22:38:21.395952-07:00 ERROR 1584: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:38:22.491087-07:00 ERROR 1584: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:38:26.671639-07:00 ERROR 1587: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:38:47.262469-07:00 ERROR 1592: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:03.628621-07:00 ERROR 874: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:05.290150-07:00 ERROR 652: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:06.988801-07:00 ERROR 65: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:07.326275-07:00 ERROR 630: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:08.087247-07:00 ERROR 627: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:08.185917-07:00 ERROR 628: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:08.475604-07:00 ERROR 64: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:08.533996-07:00 ERROR 66: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:09.873389-07:00 ERROR 73: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:10.091624-07:00 ERROR 611: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:11.177921-07:00 ERROR 74: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:11.730784-07:00 ERROR 4: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:12.147181-07:00 ERROR 63: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:13.027131-07:00 ERROR 65: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:13.745843-07:00 ERROR 4: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:13.849523-07:00 ERROR 63: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:17.917027-07:00 ERROR 63: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:20.455469-07:00 ERROR 47: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:21.428044-07:00 ERROR 64: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:22.108752-07:00 ERROR 47: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n', '2021-03-24T22:39:22.334257-07:00 ERROR 65: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n'] 2021-03-24 22:41:55,915 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1087] Running cbcollect on node ns_1@172.23.105.42,ns_1@172.23.106.9,ns_1@172.23.123.124,ns_1@172.23.123.125 2021-03-24 22:42:05,960 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1090] ns_1@172.23.105.42,ns_1@172.23.106.9,ns_1@172.23.123.124,ns_1@172.23.123.125 - cbcollect status: True 2021-03-24 22:42:05,960 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1095] Polling active_tasks to check cbcollect status 2021-03-24 22:45:27,480 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.105.42: Copying cbcollect ZIP file to Client 2021-03-24 22:45:29,174 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.105.42 node cb collect zip copied on client : True 2021-03-24 22:45:29,174 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.9: Copying cbcollect ZIP file to Client 2021-03-24 22:45:30,513 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.9 node cb collect zip copied on client : True 2021-03-24 22:45:30,513 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.124: Copying cbcollect ZIP file to Client 2021-03-24 22:45:32,141 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.124 node cb collect zip copied on client : True 2021-03-24 22:45:32,141 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.125: Copying cbcollect ZIP file to Client 2021-03-24 22:45:34,201 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.125 node cb collect zip copied on client : True 2021-03-24 22:45:49,855 | test | INFO | MainThread | [rest_client:monitorRebalance:1461] Rebalance done. Taken 11.1579999924 seconds to complete 2021-03-24 22:45:49,881 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ERROR ====================================================================== ERROR: test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 87, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 104, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 447, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 789.345s FAILED (errors=1) During the test, Remote Connections: 173, Disconnections: 173 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 3 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_3 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.5901.ini GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec.5901', 'ini': '/tmp/testexec.5901.ini', 'conf_file': 'conf/collections/collections_rebalance.conf', 'upgrade_version': '7.0.0-4774', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_4', 'nodes_init': '4', 'GROUP': 'sdk_compression', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 4, 'sdk_compression': 'True'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-03-24 22:45:58,493 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-03-24 22:45:59,036 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-03-24 22:45:59,546 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-24 22:48:38,066 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-24 22:48:55,994 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.123.125, 172.23.123.124, 172.23.106.8, 172.23.106.6, 172.23.105.42, 172.23.106.9 initialized 2021-03-24 22:48:56,019 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-03-24 22:48:56,059 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-03-24 22:49:04,174 | test | INFO | pool-5-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 0.750375187594 | Cluster node | | 172.23.123.124 | None | | | <--- IN --- | | 172.23.106.8 | None | | | <--- IN --- | | 172.23.106.6 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-03-24 22:49:04,197 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:49:14,279 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:49:14,311 | test | INFO | pool-5-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.1380000114 sec 2021-03-24 22:49:14,549 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-03-24 22:49:21,760 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2261] Waiting for all collections to be created 2021-03-24 22:49:22,026 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 419430400 | 95607536 | 16942858 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 419430400 | 76739904 | 68 | | default | couchbase | 3 | none | 0 | 0 | 6291456000 | 136667912 | 2880757 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-03-24 22:49:22,028 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-24 22:50:14,457 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:50:14,470 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:50:28,359 | test | INFO | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...................*......................*...........................................................**X | 1707.100 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:50:29,457 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....*..............*...........................................***.......................................X | 24711.000 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 22:50:29,882 | test | INFO | pool-5-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.....................*..........................*.....................................................***X | 2881.100 | +-------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:50:29,992 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.123.124 | kv | 80.8004052685 | 4201672704 | 3182489600 | 43253760 / 3758092288 | 65606 / 196036 | 7.0.0-4774-enterprise | | 172.23.106.6 | kv | 54.6693589096 | 4201840640 | 3152625664 | 0 / 3758092288 | 45611 / 135768 | 7.0.0-4774-enterprise | | 172.23.106.8 | kv | 44.7848101266 | 4201840640 | 3214548992 | 0 / 3758092288 | 131231 / 393582 | 7.0.0-4774-enterprise | | 172.23.123.125 | kv | 80.5992889792 | 4201672704 | 3100991488 | 79953920 / 3758092288 | 65861 / 195621 | 7.0.0-4774-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-24 22:50:37,746 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:51:20,588 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 419430400 | 147953760 | 110967574 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 419430400 | 123732512 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 691550048 | 529704493 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-24 22:51:20,592 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_rebalance_in_out ========= 2021-03-24 22:51:25,335 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:822] Doing collection data load during rebalance_in_out 2021-03-24 22:51:25,338 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:303] Starting rebalance operation of type : rebalance_in_out 2021-03-24 22:51:31,628 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 22:51:31,641 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 22:51:31,798 | test | INFO | pool-5-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.106.9 | kv | 7.0.0-4774-enterprise | 0.502134069797 | Cluster node | | 172.23.123.124 | kv | 7.0.0-4774-enterprise | 7.98287584991 | Cluster node | | 172.23.106.6 | kv | 7.0.0-4774-enterprise | 10.541021082 | --- OUT ---> | | 172.23.106.8 | kv | 7.0.0-4774-enterprise | 10.5730223124 | Cluster node | | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 9.35378425949 | Cluster node | | 172.23.105.42 | kv | 7.0.0-4774-enterprise | 0.376317109885 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-03-24 22:51:31,826 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 22:51:37,095 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.68 2021-03-24 22:51:42,276 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 5.89 2021-03-24 22:51:47,872 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 7.87 2021-03-24 22:51:53,125 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 10.31 2021-03-24 22:51:54,242 | test | INFO | pool-5-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *****...***..............................*...................................................................X | 1259.188 | +-------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:51:55,530 | test | INFO | pool-5-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****.***.............*.......................*...............................................................X | 19656.675 | +-------+----------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 22:51:56,861 | test | INFO | pool-5-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****...***..........................*........................*...............................................*X | 2146.624 | +-------+-----------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 22:51:58,270 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 15.03 2021-03-24 22:52:03,368 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 21.32 2021-03-24 22:52:08,441 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.66 2021-03-24 22:52:13,546 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 30.54 2021-03-24 22:52:18,621 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 22:52:23,697 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 22:52:31,411 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 22:52:36,934 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 42.98 2021-03-24 22:52:42,036 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 49.66 2021-03-24 22:52:47,161 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 56.2 2021-03-24 22:52:52,298 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 61.41 2021-03-24 22:52:57,398 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.05 2021-03-24 22:53:02,466 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 22:53:07,536 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 22:53:12,614 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.17 2021-03-24 22:53:17,710 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 77.36 2021-03-24 22:53:22,825 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 84.53 2021-03-24 22:53:27,937 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 90.31 2021-03-24 22:53:33,017 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 95.79 2021-03-24 22:53:38,108 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-03-24 22:53:43,184 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-03-24 22:53:53,315 | test | INFO | pool-5-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 22:53:53,413 | test | INFO | pool-5-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 141.61500001 sec 2021-03-24 22:54:03,127 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 22:54:54,611 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-03-24 22:54:54,611 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-03-24 22:54:55,466 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-03-24 22:54:55,467 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-03-24 22:54:56,372 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: default, Active Resident ratio(DGM): 100% 2021-03-24 22:54:56,374 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: default, Replica Resident ratio(DGM): 100% 2021-03-24 22:55:10,223 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:13:18, Test's start timestamp is 2021-03-24 22:45:58.492000 2021-03-24 22:55:11,273 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:51:36, Test's start timestamp is 2021-03-24 22:45:58.492000 2021-03-24 22:55:11,275 | test | INFO | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000013.txt 2021-03-24 22:55:11,276 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:791] 172.23.123.124: Found ' ERROR ' logs - ['2021-03-24T22:51:36.279052-07:00 ERROR 1585: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n'] 2021-03-24 22:55:14,463 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1087] Running cbcollect on node ns_1@172.23.105.42,ns_1@172.23.106.8,ns_1@172.23.106.9,ns_1@172.23.123.124,ns_1@172.23.123.125 2021-03-24 22:55:24,536 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1090] ns_1@172.23.105.42,ns_1@172.23.106.8,ns_1@172.23.106.9,ns_1@172.23.123.124,ns_1@172.23.123.125 - cbcollect status: True 2021-03-24 22:55:24,536 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1095] Polling active_tasks to check cbcollect status 2021-03-24 22:58:35,714 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.105.42: Copying cbcollect ZIP file to Client 2021-03-24 22:58:37,996 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.105.42 node cb collect zip copied on client : True 2021-03-24 22:58:37,997 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.8: Copying cbcollect ZIP file to Client 2021-03-24 22:58:40,266 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.8 node cb collect zip copied on client : True 2021-03-24 22:58:40,269 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.9: Copying cbcollect ZIP file to Client 2021-03-24 22:58:42,428 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.9 node cb collect zip copied on client : True 2021-03-24 22:58:42,430 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.124: Copying cbcollect ZIP file to Client 2021-03-24 22:58:47,098 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.124 node cb collect zip copied on client : True 2021-03-24 22:58:47,098 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.125: Copying cbcollect ZIP file to Client 2021-03-24 22:58:50,183 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.125 node cb collect zip copied on client : True 2021-03-24 22:59:06,385 | test | INFO | MainThread | [rest_client:monitorRebalance:1461] Rebalance done. Taken 11.1679999828 seconds to complete 2021-03-24 22:59:06,404 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ERROR ====================================================================== ERROR: test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 87, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 104, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 447, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 796.561s FAILED (errors=1) During the test, Remote Connections: 189, Disconnections: 189 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 4 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_4 Logs will be stored at /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.5901.ini GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,step_count=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,sdk_compression=True,skip_validations=False,GROUP=sdk_compression' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec.5901', 'ini': '/tmp/testexec.5901.ini', 'step_count': '1', 'conf_file': 'conf/collections/collections_rebalance.conf', 'upgrade_version': '7.0.0-4774', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance', 'rerun': 'False', 'num_nodes': 6, 'logs_folder': '/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_5', 'nodes_init': '3', 'GROUP': 'sdk_compression', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests', 'case_number': 5, 'sdk_compression': 'True'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-03-24 22:59:16,747 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionsRebalance setup started for test #5 test_data_load_collections_with_rebalance_in ========= 2021-03-24 22:59:17,298 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_rebalance_in ========= 2021-03-24 22:59:17,921 | test | INFO | MainThread | [basetestcase:setUp:318] Initializing cluster 2021-03-24 23:01:50,878 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:587] RAM quota was defined less than 100 MB: 2021-03-24 23:02:07,394 | test | INFO | MainThread | [basetestcase:setUp:340] Couchbase Cluster: C1, Nodes: 172.23.123.125, 172.23.123.124, 172.23.106.8, 172.23.106.6, 172.23.105.42, 172.23.106.9 initialized 2021-03-24 23:02:07,417 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= BaseTestCase setup finished for test #5 test_data_load_collections_with_rebalance_in ========= 2021-03-24 23:02:07,454 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup started for test #5 test_data_load_collections_with_rebalance_in ========= 2021-03-24 23:02:12,877 | test | INFO | pool-6-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 1.52805611222 | Cluster node | | 172.23.123.124 | None | | | <--- IN --- | | 172.23.106.8 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-24 23:02:12,904 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 23:02:22,982 | test | INFO | pool-6-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 23:02:23,006 | test | INFO | pool-6-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 10.1289999485 sec 2021-03-24 23:02:23,233 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-03-24 23:02:30,345 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2261] Waiting for all collections to be created 2021-03-24 23:02:30,586 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 314572800 | 32000472 | 0 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 314572800 | 19437064 | 0 | | default | couchbase | 3 | none | 0 | 0 | 4718592000 | 54564704 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-03-24 23:02:30,588 | test | INFO | MainThread | [collections_base:collection_setup:146] Creating required SDK clients for client_pool 2021-03-24 23:03:10,980 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 23:03:10,993 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 23:03:23,867 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****..................................**..................................................................X | 20285.800 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 23:03:24,578 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *****.............................**.......................................................................X | 2833.200 | +-------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 23:03:25,121 | test | INFO | pool-6-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****.................................**...................................................................*X | 3000.000 | +-------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 23:03:25,211 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ | 172.23.123.124 | kv | 92.951653944 | 4201672704 | 3157053440 | 43253760 / 3758092288 | 154684 / 305988 | 7.0.0-4774-enterprise | | 172.23.106.8 | kv | 74.2385786802 | 4201840640 | 3135475712 | 0 / 3758092288 | 87924 / 174727 | 7.0.0-4774-enterprise | | 172.23.123.125 | kv | 91.7707006369 | 4201672704 | 3101114368 | 79953920 / 3758092288 | 153065 / 305718 | 7.0.0-4774-enterprise | +----------------+----------+-----------------+------------+------------+-----------------------+-------------------+-----------------------+ 2021-03-24 23:03:31,316 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 23:04:00,506 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 30000 | 314572800 | 122282424 | 67655735 | | bucket2 | ephemeral | 3 | none | 0 | 30000 | 314572800 | 74606840 | 102 | | default | couchbase | 3 | none | 0 | 500000 | 4718592000 | 441556344 | 409480546 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-03-24 23:04:00,509 | test | INFO | MainThread | [basetestcase:log_setup_status:540] ========= CollectionBase setup complete for test #5 test_data_load_collections_with_rebalance_in ========= 2021-03-24 23:04:04,102 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:822] Doing collection data load during rebalance_in 2021-03-24 23:04:04,104 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:303] Starting rebalance operation of type : rebalance_in 2021-03-24 23:04:07,434 | test | INFO | pool-6-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.123.124 | kv | 7.0.0-4774-enterprise | 10.1134930643 | Cluster node | | 172.23.106.8 | kv | 7.0.0-4774-enterprise | 12.9229203765 | Cluster node | | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 12.638469285 | Cluster node | | 172.23.106.6 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-24 23:04:07,457 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 23:04:12,519 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 3.55 2021-03-24 23:04:17,576 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 10.24 2021-03-24 23:04:22,644 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 16.95 2021-03-24 23:04:27,710 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 23.82 2021-03-24 23:04:32,783 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 29.9 2021-03-24 23:04:39,059 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 23:04:44,134 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 41.13 2021-03-24 23:04:49,229 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 48.1 2021-03-24 23:04:54,309 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 54.92 2021-03-24 23:04:59,380 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 61.73 2021-03-24 23:05:04,444 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-03-24 23:05:09,529 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 72.28 2021-03-24 23:05:14,599 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 77.01 2021-03-24 23:05:19,680 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 81.93 2021-03-24 23:05:24,759 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 86.89 2021-03-24 23:05:29,826 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 91.82 2021-03-24 23:05:34,904 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: running, progress: 97.21 2021-03-24 23:05:44,993 | test | INFO | pool-6-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 23:05:45,026 | test | INFO | pool-6-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 97.5939998627 sec 2021-03-24 23:05:45,078 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4657] Performing scope/collection specific operations 2021-03-24 23:05:45,089 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4747] Done Performing scope/collection specific operations 2021-03-24 23:05:48,043 | test | INFO | pool-6-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.123.124 | kv | 7.0.0-4774-enterprise | 48.942834769 | Cluster node | | 172.23.106.6 | kv | 7.0.0-4774-enterprise | 67.3236514523 | Cluster node | | 172.23.106.8 | kv | 7.0.0-4774-enterprise | 62.3430962343 | Cluster node | | 172.23.123.125 | kv | 7.0.0-4774-enterprise | 66.363400052 | Cluster node | | 172.23.105.42 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-03-24 23:05:48,127 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-03-24 23:05:53,313 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 2.44 2021-03-24 23:05:58,509 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 5.29 2021-03-24 23:06:00,480 | test | INFO | pool-6-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.......*.............................*...............................................................***X | 11702.539 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-03-24 23:06:01,755 | test | INFO | pool-6-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.....................*..........................*..........................*...........................*X | 1350.816 | | 1295.016 | ..............................................................................................X | 1295.016 | | 1145.216 | ...................................................................................X | 1145.216 | +----------+------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 23:06:02,200 | test | INFO | pool-6-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...*......................*.......................*.......................***.............................X | 1253.268 | +-------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-03-24 23:06:03,625 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 10.56 2021-03-24 23:06:08,720 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 18.41 2021-03-24 23:06:13,822 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 25.27 2021-03-24 23:06:18,910 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-03-24 23:06:23,992 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 38.99 2021-03-24 23:06:29,082 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 47.63 2021-03-24 23:06:34,171 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 56.3 2021-03-24 23:06:39,267 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 65.01 2021-03-24 23:06:44,411 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.09 2021-03-24 23:06:49,500 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 77.63 2021-03-24 23:06:54,609 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 86.35 2021-03-24 23:06:59,703 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: running, progress: 94.43 2021-03-24 23:07:09,811 | test | INFO | pool-6-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-03-24 23:07:09,855 | test | INFO | pool-6-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 81.8120000362 sec 2021-03-24 23:07:18,552 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4411] Validating collection stats and item counts 2021-03-24 23:08:05,187 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-03-24 23:08:05,188 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-03-24 23:08:05,924 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-03-24 23:08:05,924 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-03-24 23:08:06,663 | test | INFO | MainThread | [collections_base:tearDown:92] Bucket: default, Active Resident ratio(DGM): 100% 2021-03-24 23:08:06,664 | test | INFO | MainThread | [collections_base:tearDown:95] Bucket: default, Replica Resident ratio(DGM): 100% 2021-03-24 23:08:19,650 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:13:18, Test's start timestamp is 2021-03-24 22:59:16.747000 2021-03-24 23:08:20,710 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:51:36, Test's start timestamp is 2021-03-24 22:59:16.747000 2021-03-24 23:08:20,805 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 22:53:09, Test's start timestamp is 2021-03-24 22:59:16.747000 2021-03-24 23:08:21,158 | test | INFO | MainThread | [basetestcase:check_if_new_messages:691] Comparing timestamps: Log's latest timestamp: 2021-03-24 23:06:20, Test's start timestamp is 2021-03-24 22:59:16.747000 2021-03-24 23:08:21,160 | test | INFO | MainThread | [basetestcase:check_coredump_exist:789] unwanted messages in /opt/couchbase/var/lib/couchbase/logs/memcached.log.000018.txt 2021-03-24 23:08:21,160 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:791] 172.23.123.124: Found ' ERROR ' logs - ['2021-03-24T23:06:20.403162-07:00 ERROR 1589: Connection::event_callback: unrecoverable error encountered: ["reading","error"], shutting down connection\n'] 2021-03-24 23:08:24,051 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1087] Running cbcollect on node ns_1@172.23.105.42,ns_1@172.23.106.6,ns_1@172.23.106.8,ns_1@172.23.123.124,ns_1@172.23.123.125 2021-03-24 23:08:34,095 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1090] ns_1@172.23.105.42,ns_1@172.23.106.6,ns_1@172.23.106.8,ns_1@172.23.123.124,ns_1@172.23.123.125 - cbcollect status: True 2021-03-24 23:08:34,096 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1095] Polling active_tasks to check cbcollect status 2021-03-24 23:12:05,226 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.105.42: Copying cbcollect ZIP file to Client 2021-03-24 23:12:07,793 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.105.42 node cb collect zip copied on client : True 2021-03-24 23:12:07,795 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.6: Copying cbcollect ZIP file to Client 2021-03-24 23:12:10,681 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.6 node cb collect zip copied on client : True 2021-03-24 23:12:10,683 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.106.8: Copying cbcollect ZIP file to Client 2021-03-24 23:12:13,930 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.106.8 node cb collect zip copied on client : True 2021-03-24 23:12:13,930 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.124: Copying cbcollect ZIP file to Client 2021-03-24 23:12:16,836 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.124 node cb collect zip copied on client : True 2021-03-24 23:12:16,836 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1122] ns_1@172.23.123.125: Copying cbcollect ZIP file to Client 2021-03-24 23:12:26,457 | test | ERROR | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1143] 172.23.123.125 node cb collect zip copied on client : True 2021-03-24 23:12:43,608 | test | INFO | MainThread | [rest_client:monitorRebalance:1461] Rebalance done. Taken 11.1889998913 seconds to complete 2021-03-24 23:12:43,632 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ERROR ====================================================================== ERROR: test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/collections_rebalance.py", line 87, in tearDown super(CollectionsRebalance, self).tearDown() File "pytests/bucket_collections/collections_base.py", line 104, in tearDown super(CollectionBase, self).tearDown() File "pytests/basetestcase.py", line 447, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") During the test, AssertionError: Cb_log file validation failed Remote Connections: 173, Disconnections: 173 SDK Connections: 30, Disconnections: 30 ---------------------------------------------------------------------- Ran 1 test in 817.279s FAILED (errors=1) summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 5 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in testrunner logs, diags and results are available under /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_22-07-47/test_5 Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_ephemeral,GROUP=durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_majority_and_persist_active' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load1,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_with_ttl,data_load_spec=ttl_load,data_load_stage=during,skip_validations=False,GROUP=collections_maxt_ttl_durability_persist_to_majority' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P0' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=0,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=0,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=replica_update_rebalance_P0' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P0' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P0' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,update_replica=True,updated_num_replicas=0,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,update_replica=True,updated_num_replicas=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P0' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=after,scrape_interval=2,rebalance_moves_per_node=64,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=0,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_type=sync,data_load_stage=during,skip_validations=False,GROUP=replica_update_rebalance_P1' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,change_ram_quota_cluster=True,nodes_init=3,nodes_in=2,vbuckets=64,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,change_ram_quota_cluster=True,nodes_init=5,nodes_out=2,vbuckets=128,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,change_ram_quota_cluster=True,nodes_init=4,nodes_swap=2,vbuckets=256,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_rebalance' skipped, GROUP not satisfied Test 'bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,change_ram_quota_cluster=True,nodes_init=4,nodes_in=2,vbuckets=512,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests,data_load_stage=during,skip_validations=False,GROUP=cluster_ram_update_during_rebalance' skipped, GROUP not satisfied Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 1h 5m 18s 2 actionable tasks: 2 executed workspace is /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1 fails is 5 5 Desc1: 7.0.0-4774 - collections rebalance_sdk_compression_7.0_P1 - centos (0/5) > Configure project : Using Transaction_client :: 1.1.4 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py 7.0.0-4774 --executor_jenkins_job --run_params=GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py 7.0.0-4774 --executor_jenkins_job --run_params=GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py 7.0.0-4774 --executor_jenkins_job --run_params=GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py 7.0.0-4774 --executor_jenkins_job --run_params=GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 Running: /opt/jython/bin/jython -J-cp /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/classes/java/main:/data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.4/bb27634a406d547ec910999434a483048000d266/couchbase-transactions-1.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.0/8c3109610eaf4755494582c88cc8c38333ae0c0f/java-client-3.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.0/6a41cffed1e58bdd16c554fff1a58cd52e075c0e/core-io-2.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.0/683c9c676c438e5945b0f12808575f22160c5e54/reactor-core-3.4.0.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py 7.0.0-4774 --executor_jenkins_job --run_params=GROUP=sdk_compression,rerun=False,upgrade_version=7.0.0-4774 > Task :compileJava UP-TO-DATE > Task :rerun_job Mar 24, 2021 11:13:01 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info INFO: [com.couchbase.core][CoreCreatedEvent] {"clientVersion":"3.1.0","clientGitHash":"9d504d25","coreVersion":"2.1.0","coreGitHash":"9d504d25","userAgent":"couchbase-java/3.1.0 (Linux 3.10.0-514.21.1.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_275-b01)","maxNumRequestsInRetry":32768,"ioEnvironment":{"nativeIoEnabled":true,"eventLoopThreadCount":4,"eventLoopGroups":["EpollEventLoopGroup"]},"ioConfig":{"captureTraffic":[],"mutationTokensEnabled":true,"networkResolution":"auto","dnsSrvEnabled":true,"tcpKeepAlivesEnabled":true,"tcpKeepAliveTimeMs":60000,"configPollIntervalMs":2500,"kvCircuitBreakerConfig":"disabled","queryCircuitBreakerConfig":"disabled","viewCircuitBreakerConfig":"disabled","searchCircuitBreakerConfig":"disabled","analyticsCircuitBreakerConfig":"disabled","managerCircuitBreakerConfig":"disabled","numKvConnections":1,"maxHttpConnections":12,"idleHttpConnectionTimeoutMs":4500,"configIdleRedialTimeoutMs":300000},"compressionConfig":{"enabled":true,"minRatio":0.83,"minSize":32},"securityConfig":{"tlsEnabled":false,"nativeTlsEnabled":true,"hostnameVerificationEnabled":true,"hasTrustCertificates":false,"trustManagerFactory":null},"timeoutConfig":{"kvMs":2500,"kvDurableMs":10000,"managementMs":75000,"queryMs":75000,"viewMs":75000,"searchMs":75000,"analyticsMs":75000,"connectMs":10000,"disconnectMs":10000},"loggerConfig":{"customLogger":null,"fallbackToConsole":false,"disableSlf4j":false,"loggerName":"CouchbaseLogger","diagnosticContextEnabled":false},"orphanReporterConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024},"thresholdRequestTracerConfig":{"emitIntervalMs":10000,"sampleSize":10,"queueLength":1024,"kvThresholdMs":500,"queryThresholdMs":1000,"searchThresholdMs":1000,"analyticsThresholdMs":1000,"viewThresholdMs":1000},"aggregatingMeterConfig":{"enabled":false,"emitIntervalMs":600000},"retryStrategy":"BestEffortRetryStrategy","requestTracer":"ThresholdRequestTracer","meter":"NoopMeter"} {"coreId":"0x1a61d43c00000001","seedNodes":[{"address":"172.23.121.84"}]} Mar 24, 2021 11:13:01 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0x1a61d43c00000001","managerPort":"8091","remote":"172.23.121.84"} Mar 24, 2021 11:13:02 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0x1a61d43c00000001","managerPort":"8091","remote":"172.23.104.153"} Mar 24, 2021 11:13:02 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0x1a61d43c00000001","managerPort":"8091","remote":"172.23.109.245"} Mar 24, 2021 11:13:02 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0x1a61d43c00000001","managerPort":"8091","remote":"172.23.98.63"} Mar 24, 2021 11:13:02 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info INFO: [com.couchbase.node][NodeConnectedEvent] Node connected {"coreId":"0x1a61d43c00000001","managerPort":"8091","remote":"172.23.99.54"} Mar 24, 2021 11:13:02 PM com.couchbase.client.core.cnc.LoggingEventConsumer$JdkLogger info INFO: [com.couchbase.core][BucketOpenedEvent][670ms] Opened bucket "rerun_jobs" {"coreId":"0x1a61d43c00000001"} reading from centos_collections_rebalance_sdk_compression_7.0_P1_7.0.0-4774 com.couchbase.client.core.error.DocumentNotFoundException: Document with the given id not found {"completed":true,"coreId":"0x1a61d43c00000001","idempotent":true,"lastChannelId":"1A61D43C00000001/00000000C32986E9","lastDispatchedFrom":"172.23.120.228:46696","lastDispatchedTo":"172.23.121.84:11210","requestId":39,"requestType":"GetRequest","retried":0,"service":{"bucket":"rerun_jobs","collection":"_default","documentId":"centos_collections_rebalance_sdk_compression_7.0_P1_7.0.0-4774","opaque":"0x47","scope":"_default","type":"kv"},"status":"NOT_FOUND","timeoutMs":2500,"timings":{"dispatchMicros":1866,"totalMicros":15633,"serverMicros":0}} upserted centos_collections_rebalance_sdk_compression_7.0_P1_7.0.0-4774 INFO:merge_reports:Merging of report files from logs/**/*.xml INFO:merge_reports:-- logs/testrunner-21-Mar-24_22-07-47/report-21-Mar-24_22-07-47-bucket_collections.collections_rebalance.CollectionsRebalance.xml -- INFO:merge_reports: Number of TestSuites=1 INFO:merge_reports: TestSuite#1) bucket_collections.collections_rebalance.CollectionsRebalance, Number of Tests=5 INFO:merge_reports:Summary file is at /data/workspace/centos-p0-collections-vset00-00-rebalance_sdk_compression_7.0_P1/logs/testrunner-21-Mar-24_23-13-12/merged_summary/mergedreport-21-Mar-24_23-13-12-bucket_collections.collections_rebalance.CollectionsRebalance.xml summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 0 , fail 5 failures so far... bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in No more failed tests. Stopping reruns Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 24s 2 actionable tasks: 1 executed, 1 up-to-date [description-setter] Description set: 7.0.0-4774 - collections rebalance_sdk_compression_7.0_P1 - centos (0/5) [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'propfile' [EnvInject] - Variables injected successfully. Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) #102731 is still in progress; ignoring for purposes of comparison Request made to compress build log #102731 is still in progress; ignoring for purposes of comparison Sending email to: sumedh.basarkod@couchbase.com balakumaran.gopal@couchbase.com Triggering a new build of savejoblogs Triggering a new build of test-executor-cleanup Triggering a new build of test-executor-cleanup-dynvm Finished: UNSTABLE