Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s83002 (deb12_executor) in workspace /data/workspace/temp_centos7_4node_jython Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/temp_centos7_4node_jython # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master_py3_dev^{commit} # timeout=10 Checking out Revision 0bb695760939c27a37c819c78736b645a81bf82f (origin/master_py3_dev) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 0bb695760939c27a37c819c78736b645a81bf82f > /usr/bin/git rev-list 0bb695760939c27a37c819c78736b645a81bf82f # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #3611 jenkins-temp_centos7_4node_jython-3611 # timeout=10 No emails were triggered. [temp_centos7_4node_jython] $ /bin/bash -x /tmp/jenkins167392453874549592.sh + ini_file=node.ini + echo ' [global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 # data_path:/data [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 #5:_5 #6:_6 #7:_7 #8:_8 [_1] ip:172.23.108.67 [_2] ip:172.23.108.68 [_3] ip:172.23.108.69 [_4] ip:172.23.108.70 ' + echo 'rebalance_new.rebalance_in.RebalanceInTests: rebalance_in_with_compaction_and_ops,nodes_init=3,nodes_in=1,replicas=1,num_items=100000,doc_ops=create:update:delete # test_rebalance_in_with_ops,nodes_in=3 ' + git submodule init Submodule 'couchbase_utils/cb_server_rest_util' (https://github.com/couchbaselabs/cb_server_rest_util) registered for path 'couchbase_utils/cb_server_rest_util' Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI' Submodule 'lib/goldfishAPI' (https://github.com/couchbaselabs/GoldfishRESTAPIs) registered for path 'lib/goldfishAPI' Submodule 'platform_utils/ssh_util' (https://github.com/couchbaselabs/ssh_util) registered for path 'platform_utils/ssh_util' Submodule 'py_constants' (https://github.com/couchbaselabs/py_constants.git) registered for path 'py_constants' Submodule 'sirius' (https://github.com/couchbaselabs/sirius.git) registered for path 'sirius' + git submodule update --init --force --remote Cloning into '/data/workspace/temp_centos7_4node_jython/couchbase_utils/cb_server_rest_util'... Cloning into '/data/workspace/temp_centos7_4node_jython/lib/capellaAPI'... Cloning into '/data/workspace/temp_centos7_4node_jython/lib/goldfishAPI'... Cloning into '/data/workspace/temp_centos7_4node_jython/platform_utils/ssh_util'... Cloning into '/data/workspace/temp_centos7_4node_jython/py_constants'... Cloning into '/data/workspace/temp_centos7_4node_jython/sirius'... Submodule path 'couchbase_utils/cb_server_rest_util': checked out '80deca88b5d1ca162200443f35b40047bcd3063c' Submodule path 'lib/capellaAPI': checked out '052e54d24b980276bd3dabaa622e3341eb2205c1' Submodule path 'lib/goldfishAPI': checked out 'd7811b8abc0032732076b12ec41f5cf3e381b5c5' Submodule path 'platform_utils/ssh_util': checked out '677659effa51f309c259f0fe9f8d34cd6b7c5544' Submodule path 'py_constants': checked out 'fc070c66e2d740fc6bd24f6a3ccf34438ca474ea' Submodule path 'sirius': checked out '8cba0bab1d7b6096b5c2272f139d3b2d82c208ee' [temp_centos7_4node_jython] $ /bin/bash -x /tmp/jenkins2846425192259737267.sh + '[' 1 == 1 ']' + export PATH=/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + PATH=/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + pyenv local 3.10.14 + pip install -r requirements.txt Requirement already satisfied: beautifulsoup4 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (4.12.3) Requirement already satisfied: paramiko in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (3.4.0) Requirement already satisfied: requests in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (2.25.0) Requirement already satisfied: PyYAML in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 4)) (6.0.1) Requirement already satisfied: six in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (1.15.0) Requirement already satisfied: futures in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (3.0.5) Requirement already satisfied: certifi==2023.5.7 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (2023.5.7) Requirement already satisfied: couchbase==4.2.0 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (4.2.0) Requirement already satisfied: soupsieve>1.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from beautifulsoup4->-r requirements.txt (line 1)) (2.5) Requirement already satisfied: pynacl>=1.5 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (1.5.0) Requirement already satisfied: bcrypt>=3.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (4.1.2) Requirement already satisfied: cryptography>=3.3 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (42.0.5) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (1.26.18) Requirement already satisfied: idna<3,>=2.5 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (2.10) Requirement already satisfied: chardet<4,>=3.0.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (3.0.4) Requirement already satisfied: cffi>=1.12 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from cryptography>=3.3->paramiko->-r requirements.txt (line 2)) (1.16.0) Requirement already satisfied: pycparser in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=3.3->paramiko->-r requirements.txt (line 2)) (2.22) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 24.0 [notice] To update, run: pip install --upgrade pip + '[' true == true ']' + cd platform_utils/ssh_util + export PYTHONPATH=../../couchbase_utils:../../py_constants + PYTHONPATH=../../couchbase_utils:../../py_constants + python -m install_util.install -i ../../node.ini -v 7.6.0-2182 --skip_local_download INFO: Node health check INFO: SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.67 as root INFO: os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True INFO: 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 INFO: SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.67 as root INFO: SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.68 as root INFO: os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True INFO: 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 INFO: SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.68 as root INFO: SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.69 as root INFO: os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True INFO: 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 INFO: SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.69 as root INFO: SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.70 as root INFO: os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True INFO: 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 INFO: SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.70 as root INFO: SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.67 as root INFO: 172.23.108.67 - Running 'populate_build_url' INFO: 172.23.108.67 - Build url :: http://172.23.126.166/builds/latestbuilds/couchbase-server/trinity/2182/couchbase-server-enterprise_7.6.0-2182-linux_amd64.deb INFO: SSH Connected to 172.23.108.69 as root INFO: 172.23.108.69 - Running 'populate_build_url' INFO: 172.23.108.69 - Build url :: http://172.23.126.166/builds/latestbuilds/couchbase-server/trinity/2182/couchbase-server-enterprise_7.6.0-2182-linux_amd64.deb INFO: SSH Connected to 172.23.108.68 as root INFO: 172.23.108.68 - Running 'populate_build_url' INFO: 172.23.108.68 - Build url :: http://172.23.126.166/builds/latestbuilds/couchbase-server/trinity/2182/couchbase-server-enterprise_7.6.0-2182-linux_amd64.deb INFO: SSH Connected to 172.23.108.70 as root INFO: 172.23.108.70 - Running 'populate_build_url' INFO: 172.23.108.70 - Build url :: http://172.23.126.166/builds/latestbuilds/couchbase-server/trinity/2182/couchbase-server-enterprise_7.6.0-2182-linux_amd64.deb INFO: SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.69 as root INFO: SSH Connected to 172.23.108.68 as root INFO: 172.23.108.69 - Running 'check_url_status' INFO: 172.23.108.68 - Running 'check_url_status' INFO: SSH Connected to 172.23.108.67 as root INFO: 172.23.108.67 - Running 'check_url_status' INFO: SSH Connected to 172.23.108.70 as root INFO: 172.23.108.70 - Running 'check_url_status' INFO: SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.67 as root INFO: 172.23.108.67 - Running 'download_build' INFO: SSH Connected to 172.23.108.68 as root INFO: 172.23.108.68 - Running 'download_build' INFO: SSH Connected to 172.23.108.69 as root INFO: 172.23.108.69 - Running 'download_build' INFO: SSH Connected to 172.23.108.70 as root INFO: 172.23.108.70 - Running 'download_build' INFO: Starting installation tasks :: ['uninstall', 'install', 'init_cluster', 'post_install_cleanup'] INFO: SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 INFO: SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 INFO: SSH Connected to 172.23.108.68 as root INFO: 172.23.108.68 - Running 'uninstall' INFO: SSH Connected to 172.23.108.70 as root INFO: 172.23.108.70 - Running 'uninstall' INFO: SSH Connected to 172.23.108.67 as root INFO: 172.23.108.67 - Running 'uninstall' INFO: SSH Connected to 172.23.108.69 as root INFO: 172.23.108.69 - Running 'uninstall' INFO: command executed with root but got an error ['dpkg: error: --purge needs at least one package name argument', '', 'Type dpkg --help for help about installing and deinstalling packages [*];', "Use 'apt' or 'aptitude' for user-friendly package management;", 'Type dpkg -Dhelp for a list of dpkg debug flag values;', 'Type dpkg --force-help for a list of forcing options;', 'Type dpkg-deb --help for help about manipulating *.deb files;', '', "Opt ... INFO: 172.23.108.69 - Running 'install' INFO: command executed with root but got an error ['dpkg: error: --purge needs at least one package name argument', '', 'Type dpkg --help for help about installing and deinstalling packages [*];', "Use 'apt' or 'aptitude' for user-friendly package management;", 'Type dpkg -Dhelp for a list of dpkg debug flag values;', 'Type dpkg --force-help for a list of forcing options;', 'Type dpkg-deb --help for help about manipulating *.deb files;', '', "Opt ... INFO: 172.23.108.70 - Running 'install' INFO: command executed with root but got an error ['dpkg: error: --purge needs at least one package name argument', '', 'Type dpkg --help for help about installing and deinstalling packages [*];', "Use 'apt' or 'aptitude' for user-friendly package management;", 'Type dpkg -Dhelp for a list of dpkg debug flag values;', 'Type dpkg --force-help for a list of forcing options;', 'Type dpkg-deb --help for help about manipulating *.deb files;', '', "Opt ... INFO: 172.23.108.68 - Running 'install' INFO: command executed with root but got an error ['dpkg: error: --purge needs at least one package name argument', '', 'Type dpkg --help for help about installing and deinstalling packages [*];', "Use 'apt' or 'aptitude' for user-friendly package management;", 'Type dpkg -Dhelp for a list of dpkg debug flag values;', 'Type dpkg --force-help for a list of forcing options;', 'Type dpkg-deb --help for help about manipulating *.deb files;', '', "Opt ... INFO: 172.23.108.67 - Running 'install' INFO: command executed with root but got an error ['debconf: unable to initialize frontend: Dialog', 'debconf: (TERM is not set, so the dialog frontend is not usable.)', 'debconf: falling back to frontend: Readline', 'debconf: unable to initialize frontend: Readline', 'debconf: (This frontend requires a controlling tty.)', 'debconf: falling back to frontend: Teletype', 'dpkg-preconfigure: unable to re-open stdin: '] ... INFO: command executed with root but got an error ['debconf: unable to initialize frontend: Dialog', 'debconf: (TERM is not set, so the dialog frontend is not usable.)', 'debconf: falling back to frontend: Readline', 'debconf: unable to initialize frontend: Readline', 'debconf: (This frontend requires a controlling tty.)', 'debconf: falling back to frontend: Teletype', 'dpkg-preconfigure: unable to re-open stdin: '] ... INFO: 172.23.108.70 - Running 'init_cluster' INFO: 172.23.108.69 - Running 'init_cluster' INFO: command executed with root but got an error ['debconf: unable to initialize frontend: Dialog', 'debconf: (TERM is not set, so the dialog frontend is not usable.)', 'debconf: falling back to frontend: Readline', 'debconf: unable to initialize frontend: Readline', 'debconf: (This frontend requires a controlling tty.)', 'debconf: falling back to frontend: Teletype', 'dpkg-preconfigure: unable to re-open stdin: '] ... INFO: 172.23.108.68 - Running 'init_cluster' INFO: command executed with root but got an error ['debconf: unable to initialize frontend: Dialog', 'debconf: (TERM is not set, so the dialog frontend is not usable.)', 'debconf: falling back to frontend: Readline', 'debconf: unable to initialize frontend: Readline', 'debconf: (This frontend requires a controlling tty.)', 'debconf: falling back to frontend: Teletype', 'dpkg-preconfigure: unable to re-open stdin: '] ... INFO: 172.23.108.67 - Running 'init_cluster' INFO: 172.23.108.70 - Running 'post_install_cleanup' INFO: 172.23.108.69 - Running 'post_install_cleanup' INFO: 172.23.108.68 - Running 'post_install_cleanup' INFO: 172.23.108.67 - Running 'post_install_cleanup' INFO: 172.23.108.67: Complete 172.23.108.68: Complete 172.23.108.69: Complete 172.23.108.70: Complete INFO: TOTAL INSTALL TIME = 135 seconds + status=0 + cd ../.. + port=49152 + INCR=1 ++ ss -tan4H 'sport = 49152' + '[' -n '' ']' + echo 49152 49152 + echo 'Will use 49152 for starting sirius' Will use 49152 for starting sirius + '[' 'git fetch https://review.couchbase.org/TAF refs/changes/14/209014/1 && git cherry-pick FETCH_HEAD' '!=' '' ']' + sh -c 'git fetch https://review.couchbase.org/TAF refs/changes/14/209014/1 && git cherry-pick FETCH_HEAD' From https://review.couchbase.org/TAF * branch refs/changes/14/209014/1 -> FETCH_HEAD [detached HEAD 935e5a50] Fixing bucket_name param instead of sending it as Bucket obj Author: Ashwin Date: Wed Apr 24 17:47:13 2024 +0530 2 files changed, 3 insertions(+), 3 deletions(-) + python testrunner.py -c debug.conf -i node.ini -p get-cbcollect-info=False,skip_cluster_reset=True,load_docs_using=sirius_go_sdk --launch_sirius_docker --sirius_url http://localhost:49152 rebalance_in_with_compaction_and_ops (rebalance_new.rebalance_in.RebalanceInTests) Rebalances nodes into a cluster while doing ... Filename: debug.conf Prefix: rebalance_new.rebalance_in.RebalanceInTests Global Test input params: {'cluster_name': 'node', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'ini': 'node.ini', 'load_docs_using': 'sirius_go_sdk', 'num_nodes': 4, 'skip_cluster_reset': 'True', 'spec': 'debug'} Logs will be stored at /data/workspace/temp_centos7_4node_jython/logs/testrunner-24-Apr-24_05-40-41/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini -p get-cbcollect-info=False,skip_cluster_reset=True,load_docs_using=sirius_go_sdk -t rebalance_new.rebalance_in.RebalanceInTests.rebalance_in_with_compaction_and_ops,nodes_init=3,nodes_in=1,replicas=1,num_items=100000,doc_ops=create:update:delete' Test Input params: {'nodes_init': '3', 'nodes_in': '1', 'replicas': '1', 'num_items': '100000', 'doc_ops': 'create:update:delete', 'sirius_url': 'http://localhost:49152', 'ini': 'node.ini', 'cluster_name': 'node', 'spec': 'debug', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'skip_cluster_reset': 'True', 'load_docs_using': 'sirius_go_sdk', 'num_nodes': 4, 'no_of_test_identified': 1, 'case_number': 1, 'logs_folder': '/data/workspace/temp_centos7_4node_jython/logs/testrunner-24-Apr-24_05-40-41/test_1', 'rerun': False} 2024-04-24 05:40:44,007 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:40:44,008 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:40:44,008 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:40:44,009 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 05:40:44,102 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:40:44,104 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:40:44,105 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 05:40:44,106 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:40:44,266 | shell_util | INFO | ThreadPool_2 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:40:44,268 | shell_util | INFO | ThreadPool_3 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:40:44,272 | shell_util | INFO | ThreadPool_0 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:40:44,348 | shell_util | INFO | ThreadPool_1 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:40:44,566 | shell_util | INFO | ThreadPool_0 | [shell_conn:extract_remote_info:521] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:40:44,567 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:40:44,570 | shell_util | INFO | ThreadPool_2 | [shell_conn:extract_remote_info:521] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:40:44,570 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:40:44,574 | shell_util | INFO | ThreadPool_3 | [shell_conn:extract_remote_info:521] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:40:44,575 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 05:40:44,648 | shell_util | INFO | ThreadPool_1 | [shell_conn:extract_remote_info:521] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:40:44,649 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:40:44,666 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:40:44,692 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 05:40:44,698 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:40:44,737 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:40:44,738 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= RebalanceInTests setup started for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 05:40:44,738 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:40:44,739 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:40:44,739 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:40:44,740 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 05:40:44,829 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 05:40:44,838 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:40:44,840 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:40:44,840 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:40:46,840 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:40:46,939 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:40:47,099 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:40:47,392 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:40:47,392 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:40:47,481 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:40:47,481 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= OnPremBaseTest setup started for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 05:40:47,519 | test | INFO | MainThread | [onPrem_basetestcase:setUp:270] Delete all buckets and rebalance out other nodes from 'C1' 2024-04-24 05:40:47,600 | test | WARNING | MainThread | [cluster_ready_functions:stop_running_rebalance:568] Rebalance stop was attempted but there is no ongoing rebalance 2024-04-24 05:40:48,512 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:40:48,610 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:40:50,660 | test | INFO | MainThread | [onPrem_basetestcase:enable_tls_on_nodes:535] Validating if services obey tls only on servers [ip:172.23.108.67 port:8091 ssh_username:root, ip:172.23.108.68 port:8091 ssh_username:root, ip:172.23.108.69 port:8091 ssh_username:root, ip:172.23.108.70 port:8091 ssh_username:root] 2024-04-24 05:40:50,661 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:40:50,761 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:40:51,139 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= OnPremBaseTest setup finished for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 05:40:51,193 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= OnPremBaseTest setup finished for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 05:41:14,402 | test | CRITICAL | ThreadPool_3 | [cluster_ready_functions:validate_orchestrator_selection:473] Orchestrator: 172.23.108.67 2024-04-24 05:41:14,541 | test | INFO | ThreadPool_3 | [table_view:display:72] Rebalance Overview +---------------+---------+----------+---------------------------------+-----+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+----------+---------------------------------+-----+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 7.6.0-2182-enterprise / default | 0 | Cluster node | active / none | | 172.23.108.68 | None | kv | | | <--- IN --- | | | 172.23.108.69 | None | kv | | | <--- IN --- | | +---------------+---------+----------+---------------------------------+-----+--------------+-----------------------+ 2024-04-24 05:41:14,653 | test | INFO | ThreadPool_3 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-04-24 05:41:29,765 | test | INFO | ThreadPool_3 | [task:check:1002] Rebalance - status: none, progress: 100 2024-04-24 05:41:29,816 | test | ERROR | ThreadPool_3 | [cluster_ready_functions:is_cluster_mixed:2263] Error while processing cluster info {'name': 'default', 'nodes': [{'clusterMembership': 'active', 'recoveryType': 'none', 'status': 'healthy', 'otpNode': 'ns_1@172.23.108.67', 'thisNode': True, 'hostname': '172.23.108.67:8091', 'nodeUUID': '21b0148ee48af45857cb02f43c370cfa', 'clusterCompatibility': 458758, 'version': '7.6.0-2182-enterprise', 'os': 'x86_64-pc-linux-gnu', 'cpuCount': 4, 'ports': {'direct': 11210, 'httpsMgmt': 18091, 'httpsCAPI': 18092, 'distTCP': 21100, 'distTLS': 21150}, 'services': ['kv'], 'nodeEncryption': True, 'nodeEncryptionClientCertVerification': False, 'addressFamilyOnly': False, 'configuredHostname': '172.23.108.67:8091', 'addressFamily': 'inet', 'externalListeners': [{'afamily': 'inet', 'nodeEncryption': True}], 'serverGroup': 'Group 1', 'couchApiBase': 'http://172.23.108.67:8092/', 'couchApiBaseHTTPS': 'https://172.23.108.67:18092/', 'nodeHash': 19605087, 'systemStats': {'cpu_utilization_rate': 5.100000016391277, 'cpu_stolen_rate': 0.1750000000174623, 'swap_total': 4550815744, 'swap_used': 26746880, 'mem_total': 4326473728, 'mem_free': 3448860672, 'mem_limit': 4326473728, 'cpu_cores_available': 4, 'allocstall': 823679}, 'interestingStats': {}, 'uptime': '67', 'memoryTotal': 4326473728, 'memoryFree': 3448860672, 'mcdMemoryReserved': 3300, 'mcdMemoryAllocated': 3300}, {'clusterMembership': 'active', 'recoveryType': 'none', 'status': 'healthy', 'otpNode': 'ns_1@172.23.108.68', 'hostname': '172.23.108.68:8091', 'nodeUUID': 'c88f8f843ae95e63489818ab283120a3', 'clusterCompatibility': 458758, 'version': '7.6.0-2182-enterprise', 'os': 'x86_64-pc-linux-gnu', 'cpuCount': 4, 'ports': {'direct': 11210, 'httpsMgmt': 18091, 'httpsCAPI': 18092, 'distTCP': 21100, 'distTLS': 21150}, 'services': ['kv'], 'nodeEncryption': True, 'nodeEncryptionClientCertVerification': False, 'addressFamilyOnly': False, 'configuredHostname': '172.23.108.68:8091', 'addressFamily': 'inet', 'externalListeners': [{'afamily': 'inet', 'nodeEncryption': True}], 'serverGroup': 'Group 1', 'couchApiBase': 'http://172.23.108.68:8092/', 'couchApiBaseHTTPS': 'https://172.23.108.68:18092/', 'nodeHash': 60303418, 'systemStats': {'cpu_utilization_rate': 11.28750000149012, 'cpu_stolen_rate': 0.2124999999796273, 'swap_total': 4550815744, 'swap_used': 15204352, 'mem_total': 4326473728, 'mem_free': 3398725632, 'mem_limit': 4326473728, 'cpu_cores_available': 4, 'allocstall': 666}, 'interestingStats': {}, 'uptime': '67', 'memoryTotal': 4326473728, 'memoryFree': 3398725632, 'mcdMemoryReserved': 3300, 'mcdMemoryAllocated': 3300}, {'clusterMembership': 'active', 'recoveryType': 'none', 'status': 'healthy', 'otpNode': 'ns_1@172.23.108.69', 'hostname': '172.23.108.69:8091', 'nodeUUID': 'b3cd968170d544b180630adf7fb9cae6', 'clusterCompatibility': 458758, 'version': '7.6.0-2182-enterprise', 'os': 'x86_64-pc-linux-gnu', 'cpuCount': 4, 'ports': {'direct': 11210, 'httpsMgmt': 18091, 'httpsCAPI': 18092, 'distTCP': 21100, 'distTLS': 21150}, 'services': ['kv'], 'nodeEncryption': True, 'nodeEncryptionClientCertVerification': False, 'addressFamilyOnly': False, 'configuredHostname': '172.23.108.69:8091', 'addressFamily': 'inet', 'externalListeners': [{'afamily': 'inet', 'nodeEncryption': True}], 'serverGroup': 'Group 1', 'couchApiBase': 'http://172.23.108.69:8092/', 'couchApiBaseHTTPS': 'https://172.23.108.69:18092/', 'nodeHash': 101399795, 'systemStats': {'cpu_utilization_rate': 11.57942103565375, 'cpu_stolen_rate': 0.2374881255966305, 'swap_total': 4550815744, 'swap_used': 15728640, 'mem_total': 4326473728, 'mem_free': 3381313536, 'mem_limit': 4326473728, 'cpu_cores_available': 4, 'allocstall': 161}, 'interestingStats': {}, 'uptime': '74', 'memoryTotal': 4326473728, 'memoryFree': 3381313536, 'mcdMemoryReserved': 3300, 'mcdMemoryAllocated': 3300}], 'buckets': {'uri': '/pools/default/buckets?v=109581657&uuid=3f0662467d458da052c12f804dedc0b4', 'terseBucketsBase': '/pools/default/b/', 'terseStreamingBucketsBase': '/pools/default/bs/'}, 'bucketNames': [], 'remoteClusters': {'uri': '/pools/default/remoteClusters?uuid=3f0662467d458da052c12f804dedc0b4', 'validateURI': '/pools/default/remoteClusters?just_validate=1'}, 'alerts': [], 'alertsSilenceURL': '/controller/resetAlerts?uuid=3f0662467d458da052c12f804dedc0b4&token=0', 'controllers': {'addNode': {'uri': '/controller/addNodeV2?uuid=3f0662467d458da052c12f804dedc0b4'}, 'rebalance': {'uri': '/controller/rebalance?uuid=3f0662467d458da052c12f804dedc0b4'}, 'failOver': {'uri': '/controller/failOver?uuid=3f0662467d458da052c12f804dedc0b4'}, 'startGracefulFailover': {'uri': '/controller/startGracefulFailover?uuid=3f0662467d458da052c12f804dedc0b4'}, 'reAddNode': {'uri': '/controller/reAddNode?uuid=3f0662467d458da052c12f804dedc0b4'}, 'reFailOver': {'uri': '/controller/reFailOver?uuid=3f0662467d458da052c12f804dedc0b4'}, 'ejectNode': {'uri': '/controller/ejectNode?uuid=3f0662467d458da052c12f804dedc0b4'}, 'setRecoveryType': {'uri': '/controller/setRecoveryType?uuid=3f0662467d458da052c12f804dedc0b4'}, 'setAutoCompaction': {'uri': '/controller/setAutoCompaction?uuid=3f0662467d458da052c12f804dedc0b4', 'validateURI': '/controller/setAutoCompaction?just_validate=1'}, 'clusterLogsCollection': {'startURI': '/controller/startLogsCollection?uuid=3f0662467d458da052c12f804dedc0b4', 'cancelURI': '/controller/cancelLogsCollection?uuid=3f0662467d458da052c12f804dedc0b4'}, 'replication': {'createURI': '/controller/createReplication?uuid=3f0662467d458da052c12f804dedc0b4', 'validateURI': '/controller/createReplication?just_validate=1'}}, 'rebalanceStatus': 'none', 'rebalanceProgressUri': '/pools/default/rebalanceProgress', 'stopRebalanceUri': '/controller/stopRebalance?uuid=3f0662467d458da052c12f804dedc0b4', 'nodeStatusesUri': '/nodeStatuses', 'nodeServicesUri': '/pools/default/nodeServices?v=29759942', 'maxBucketCount': 30, 'maxCollectionCount': 1200, 'maxScopeCount': 1200, 'minReplicasCount': 0, 'autoCompactionSettings': {'parallelDBAndViewCompaction': False, 'magmaFragmentationPercentage': 50, 'databaseFragmentationThreshold': {'percentage': 30, 'size': 'undefined'}, 'viewFragmentationThreshold': {'percentage': 30, 'size': 'undefined'}, 'indexCompactionMode': 'circular', 'indexCircularCompaction': {'daysOfWeek': 'Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday', 'interval': {'fromHour': 0, 'toHour': 0, 'fromMinute': 0, 'toMinute': 0, 'abortOutside': False}}, 'indexFragmentationThreshold': {'percentage': 30}}, 'tasks': {'uri': '/pools/default/tasks?v=66019051'}, 'counters': {'rebalance_success': 1, 'rebalance_start': 1}, 'indexStatusURI': '/indexStatus?v=21137658', 'trustedCAsURI': '/pools/default/trustedCAs?v=94531312', 'clusterName': '', 'clusterEncryptionLevel': 'all', 'balanced': True, 'checkPermissionsURI': '/pools/default/checkPermissions?v=4blGFP22Qv%2FFR5oU9ygPHq8Tscw%3D', 'memoryQuota': 2200, 'queryMemoryQuota': 0, 'indexMemoryQuota': 512, 'ftsMemoryQuota': 436, 'cbasMemoryQuota': 1239, 'eventingMemoryQuota': 256, 'storageTotals': {'ram': {'total': 12979421184, 'quotaTotal': 6920601600, 'quotaUsed': 0, 'used': 9957609472, 'usedByData': 0, 'quotaUsedPerNode': 0, 'quotaTotalPerNode': 2306867200}, 'hdd': {'total': 96139296768, 'quotaTotal': 96139296768, 'used': 14100430191, 'usedByData': 0, 'free': 81718402254}}, 'serverGroupsUri': '/pools/default/serverGroups?v=114066866'} 2024-04-24 05:41:29,816 | test | INFO | ThreadPool_3 | [task:check:1070] Rebalance completed with progress: 100% in 15.275837898254395 sec 2024-04-24 05:41:29,963 | test | CRITICAL | ThreadPool_3 | [cluster_ready_functions:validate_orchestrator_selection:473] Orchestrator: 172.23.108.67 2024-04-24 05:41:30,074 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:743] Cluster nodes..: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091'] 2024-04-24 05:41:30,074 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:743] KV............: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091'] 2024-04-24 05:41:30,185 | rest_api | INFO | ThreadPool_2 | [BucketOperations_Rest:create_bucket:390] Creating 'couchbase' bucket default 2024-04-24 05:41:30,247 | rest_api | DEBUG | ThreadPool_2 | [BucketOperations_Rest:create_bucket:399] 0.06 seconds to create bucket default 2024-04-24 05:41:30,288 | test | INFO | MainThread | [common_lib:sleep:21] Sleep 2 seconds. Reason: No servers_list found, Will retry... 2024-04-24 05:41:34,330 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:41:34,424 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:41:35,029 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:41:35,122 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:41:35,287 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:41:35,594 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:41:35,594 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:41:35,686 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:41:36,323 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:41:36,422 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:41:36,589 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:41:36,894 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:41:36,895 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:41:36,994 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/bulk-create /bulk-create {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 0, 'end': 100000, 'count': 100000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'insertOptions': {'expiry': 0, 'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 30}} http://localhost:49152/result /result {'seed': '1713962497779488987', 'deleteRecord': False} http://localhost:49152/result /result {'seed': '1713962497779488987', 'deleteRecord': False} 2024-04-24 05:41:49,829 | test | INFO | ThreadPool_3 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+----------+ | 4621.700 | .............................................................******......................................X | 7431.300 | +----------+------------------------------------------------------------------------------------------------------------+----------+ 2024-04-24 05:41:49,830 | test | ERROR | ThreadPool_1 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' http://localhost:49152/retry-exceptions /retry-exceptions {'identifierToken': 'UNIQUE_1713962440', 'resultSeed': '1713962497779488987', 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 1}} http://localhost:49152/result /result {'seed': '1713962497779488987', 'deleteRecord': False} http://localhost:49152/result /result {'seed': '1713962497779488987', 'deleteRecord': False} 2024-04-24 05:42:49,902 | test | INFO | MainThread | [rebalance_base:setUp:153] Verifying num_items counts after doc_ops 2024-04-24 05:42:49,945 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:42:50,042 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:42:50,962 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:42:51,124 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:42:51,953 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:42:52,064 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:42:52,995 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:5751] Validating collection stats and item counts 2024-04-24 05:42:53,059 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:42:53,155 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:42:53,519 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:42:53,617 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:42:53,994 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:42:54,092 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:42:54,467 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:42:54,467 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:42:54,560 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:42:54,560 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:42:54,588 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:42:54,588 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:42:54,660 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:42:54,660 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:42:54,705 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:42:54,705 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:42:54,752 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:42:54,800 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:42:55,354 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:42:55,451 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:42:55,452 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:42:55,549 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:42:55,549 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:42:55,647 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:42:56,115 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:42:56,211 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:42:56,211 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:42:56,296 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:42:56,296 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:42:56,393 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:42:56,566 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | 172.23.108.67 | Group 1 | kv | 4.0000 | 4.03 GiB | 3.03 GiB | 25.51 MiB / 4.24 GiB | 33401 / 33297 | 7.6.0-2182-enterprise / default | | 172.23.108.68 | Group 1 | kv | 3.6249 | 4.03 GiB | 2.99 GiB | 14.50 MiB / 4.24 GiB | 33297 / 33399 | 7.6.0-2182-enterprise / default | | 172.23.108.69 | Group 1 | kv | 3.5750 | 4.03 GiB | 3.01 GiB | 15.00 MiB / 4.24 GiB | 33302 / 33304 | 7.6.0-2182-enterprise / default | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ 2024-04-24 05:42:56,758 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+-----+ | Bucket | Type / Storage | Replicas | Rank | Vbuckets | Durability | TTL | Items | RAM Quota / Used | Disk Used | ARR | +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+-----+ | default | couchbase / magma | 1 | None | 1024 | none | 0 | 100000 | 768.00 MiB / 322.12 MiB | 239.00 MiB | 100 | +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+-----+ 2024-04-24 05:42:56,758 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= RebalanceBase setup complete for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 05:42:56,959 | test | INFO | MainThread | [common_lib:sleep:21] Sleep 10 seconds. Reason: wait for rebalance to start 2024-04-24 05:43:05,000 | test | CRITICAL | ThreadPool_0 | [cluster_ready_functions:validate_orchestrator_selection:473] Orchestrator: 172.23.108.67 2024-04-24 05:43:05,170 | test | INFO | ThreadPool_0 | [table_view:display:72] Rebalance Overview +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 7.6.0-2182-enterprise / default | 4.000000022351742 | Cluster node | active / none | | 172.23.108.68 | Group 1 | kv | 7.6.0-2182-enterprise / default | 3.624999988824129 | Cluster node | active / none | | 172.23.108.69 | Group 1 | kv | 7.6.0-2182-enterprise / default | 3.575000017881393 | Cluster node | active / none | | 172.23.108.70 | None | kv | | | <--- IN --- | | +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ 2024-04-24 05:43:05,277 | test | INFO | ThreadPool_0 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-04-24 05:43:07,028 | rest_api | DEBUG | ThreadPool_1 | [BucketOperations_Rest:compact_bucket:647] Triggering bucket compaction for 'default' http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/bulk-upsert /bulk-upsert {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 0, 'end': 50000, 'count': 50000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'insertOptions': {'expiry': 0, 'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 5}} http://localhost:49152/result /result {'seed': '1713962587055216890', 'deleteRecord': False} http://localhost:49152/bulk-create /bulk-create {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 100000, 'end': 200000, 'count': 100000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'insertOptions': {'expiry': 0, 'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 5}} 2024-04-24 05:43:07,075 | rest_api | DEBUG | ThreadPool_1 | [BucketOperations_Rest:compact_bucket:651] Bucket compaction successful http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/result /result {'seed': '1713962587094252723', 'deleteRecord': False} http://localhost:49152/bulk-delete /bulk-delete {'identifierToken': 'UNIQUE_1713962440', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 50000, 'end': 100000, 'count': 50000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'removeOptions': {'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 5}} http://localhost:49152/result /result {'seed': '1713962587126371889', 'deleteRecord': False} 2024-04-24 05:43:15,338 | test | ERROR | ThreadPool_0 | [task:call:829] 'ClusterUtils' object has no attribute 'test_log' 2024-04-24 05:43:15,394 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | 172.23.108.67 | Group 1 | kv | 10.075 | 4.03 GiB | 2.93 GiB | 25.51 MiB / 4.24 GiB | 33038 / 33091 | 7.6.0-2182-enterprise / default | | 172.23.108.68 | Group 1 | kv | 8.0000 | 4.03 GiB | 2.95 GiB | 14.50 MiB / 4.24 GiB | 32729 / 32747 | 7.6.0-2182-enterprise / default | | 172.23.108.69 | Group 1 | kv | 8.9499 | 4.03 GiB | 2.95 GiB | 15.00 MiB / 4.24 GiB | 32765 / 32921 | 7.6.0-2182-enterprise / default | | 172.23.108.70 | Group 1 | kv | 1.5500 | 4.03 GiB | 3.18 GiB | 20.75 MiB / 4.24 GiB | 0 / 0 | 7.6.0-2182-enterprise / default | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ 2024-04-24 05:43:15,395 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:43:15,516 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:43:16,927 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.67 - Parsing logs for error/critical string patterns 2024-04-24 05:43:16,928 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 05:43:17,017 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 05:43:17,156 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:43:17,456 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:43:18,253 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 05:43:18,762 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... http://localhost:49152/result /result {'seed': '1713962587055216890', 'deleteRecord': False} 2024-04-24 05:43:19,089 | test | ERROR | ThreadPool_3 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' http://localhost:49152/result /result {'seed': '1713962587094252723', 'deleteRecord': False} 2024-04-24 05:43:19,120 | test | ERROR | ThreadPool_5 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' http://localhost:49152/result /result {'seed': '1713962587126371889', 'deleteRecord': False} 2024-04-24 05:43:19,970 | test | INFO | ThreadPool_4 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...............**.....................................................................................******X | 8766.900 | +-------+-----------------------------------------------------------------------------------------------------------------+----------+ 2024-04-24 05:43:19,972 | test | ERROR | ThreadPool_7 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' 2024-04-24 05:43:26,844 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.68 - Parsing logs for error/critical string patterns 2024-04-24 05:43:26,844 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 05:43:26,925 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 05:43:27,082 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:43:27,373 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:43:28,174 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 05:43:28,720 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-04-24 05:43:36,978 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.69 - Parsing logs for error/critical string patterns 2024-04-24 05:43:36,978 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 05:43:37,071 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 05:43:37,228 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:43:37,536 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:43:38,366 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 05:43:38,929 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-04-24 05:43:47,328 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.70 - Parsing logs for error/critical string patterns 2024-04-24 05:43:47,328 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 05:43:47,423 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 05:43:47,582 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:43:47,888 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:43:47,888 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 05:43:47,979 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 05:43:48,144 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 05:43:48,459 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 05:43:49,260 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 05:43:49,850 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-04-24 05:43:58,170 | test | CRITICAL | MainThread | [onPrem_basetestcase:tearDownEverything:661] Skipping get_trace !! FAIL ====================================================================== FAIL: rebalance_in_with_compaction_and_ops (rebalance_new.rebalance_in.RebalanceInTests) Rebalances nodes into a cluster while doing ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/workspace/temp_centos7_4node_jython/pytests/rebalance_new/rebalance_in.py", line 510, in rebalance_in_with_compaction_and_ops self.fail("Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 204.205s FAILED (failures=1) During the test, Remote Connections: 0, Disconnections: 0 SDK Connections: 0, Disconnections: 0 summary so far suite rebalance_new.rebalance_in.RebalanceInTests , pass 0 , fail 1 failures so far... rebalance_new.rebalance_in.RebalanceInTests.rebalance_in_with_compaction_and_ops testrunner logs, diags and results are available under /data/workspace/temp_centos7_4node_jython/logs/testrunner-24-Apr-24_05-40-41/test_1 + exit 0 Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Could not determine description. [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Sending email to: ashwin.govindarajulu@couchbase.com Finished: UNSTABLE