Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s83002 (deb12_executor) in workspace /data/workspace/temp_centos7_4node_jython Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/temp_centos7_4node_jython # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master_py3_dev^{commit} # timeout=10 Checking out Revision 0bb695760939c27a37c819c78736b645a81bf82f (origin/master_py3_dev) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 0bb695760939c27a37c819c78736b645a81bf82f > /usr/bin/git rev-list 5df08e017d6eca8389e40ffa989c50aa17162e7e # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #3607 jenkins-temp_centos7_4node_jython-3607 # timeout=10 No emails were triggered. [temp_centos7_4node_jython] $ /bin/bash -x /tmp/jenkins123994993651123715.sh + ini_file=node.ini + echo ' [global] username:root password:couchbase port:8091 index_port:9102 n1ql_port:8093 # data_path:/data [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 #5:_5 #6:_6 #7:_7 #8:_8 [_1] ip:172.23.108.67 [_2] ip:172.23.108.68 [_3] ip:172.23.108.69 [_4] ip:172.23.108.70 ' + echo 'rebalance_new.rebalance_in.RebalanceInTests: rebalance_in_with_compaction_and_ops,nodes_init=3,nodes_in=1,replicas=1,num_items=100000,doc_ops=create:update:delete # test_rebalance_in_with_ops,nodes_in=3 ' + git submodule init Submodule 'couchbase_utils/cb_server_rest_util' (https://github.com/couchbaselabs/cb_server_rest_util) registered for path 'couchbase_utils/cb_server_rest_util' Submodule 'lib/capellaAPI' (https://github.com/couchbaselabs/CapellaRESTAPIs) registered for path 'lib/capellaAPI' Submodule 'lib/goldfishAPI' (https://github.com/couchbaselabs/GoldfishRESTAPIs) registered for path 'lib/goldfishAPI' Submodule 'platform_utils/ssh_util' (https://github.com/couchbaselabs/ssh_util) registered for path 'platform_utils/ssh_util' Submodule 'py_constants' (https://github.com/couchbaselabs/py_constants.git) registered for path 'py_constants' Submodule 'sirius' (https://github.com/couchbaselabs/sirius.git) registered for path 'sirius' + git submodule update --init --force --remote Cloning into '/data/workspace/temp_centos7_4node_jython/couchbase_utils/cb_server_rest_util'... Cloning into '/data/workspace/temp_centos7_4node_jython/lib/capellaAPI'... Cloning into '/data/workspace/temp_centos7_4node_jython/lib/goldfishAPI'... Cloning into '/data/workspace/temp_centos7_4node_jython/platform_utils/ssh_util'... Cloning into '/data/workspace/temp_centos7_4node_jython/py_constants'... Cloning into '/data/workspace/temp_centos7_4node_jython/sirius'... Submodule path 'couchbase_utils/cb_server_rest_util': checked out '80deca88b5d1ca162200443f35b40047bcd3063c' Submodule path 'lib/capellaAPI': checked out '052e54d24b980276bd3dabaa622e3341eb2205c1' Submodule path 'lib/goldfishAPI': checked out 'd7811b8abc0032732076b12ec41f5cf3e381b5c5' Submodule path 'platform_utils/ssh_util': checked out '677659effa51f309c259f0fe9f8d34cd6b7c5544' Submodule path 'py_constants': checked out 'fc070c66e2d740fc6bd24f6a3ccf34438ca474ea' Submodule path 'sirius': checked out '8cba0bab1d7b6096b5c2272f139d3b2d82c208ee' [temp_centos7_4node_jython] $ /bin/bash -x /tmp/jenkins6528358717639339483.sh + '[' 1 == 1 ']' + '[' false == true ']' + port=49152 + INCR=1 ++ ss -tan4H 'sport = 49152' + '[' -n '' ']' + echo 49152 49152 + echo 'Will use 49152 for starting sirius' Will use 49152 for starting sirius + '[' '' '!=' '' ']' + export PATH=/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + PATH=/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + pyenv local 3.10.14 + pip install -r requirements.txt Requirement already satisfied: beautifulsoup4 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (4.12.3) Requirement already satisfied: paramiko in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (3.4.0) Requirement already satisfied: requests in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (2.25.0) Requirement already satisfied: PyYAML in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 4)) (6.0.1) Requirement already satisfied: six in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (1.15.0) Requirement already satisfied: futures in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (3.0.5) Requirement already satisfied: certifi==2023.5.7 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (2023.5.7) Requirement already satisfied: couchbase==4.2.0 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (4.2.0) Requirement already satisfied: soupsieve>1.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from beautifulsoup4->-r requirements.txt (line 1)) (2.5) Requirement already satisfied: bcrypt>=3.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (4.1.2) Requirement already satisfied: cryptography>=3.3 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (42.0.5) Requirement already satisfied: pynacl>=1.5 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (1.5.0) Requirement already satisfied: chardet<4,>=3.0.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (3.0.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (1.26.18) Requirement already satisfied: idna<3,>=2.5 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (2.10) Requirement already satisfied: cffi>=1.12 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from cryptography>=3.3->paramiko->-r requirements.txt (line 2)) (1.16.0) Requirement already satisfied: pycparser in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=3.3->paramiko->-r requirements.txt (line 2)) (2.22) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 24.0 [notice] To update, run: pip install --upgrade pip + python testrunner.py -c debug.conf -i node.ini -p get-cbcollect-info=False,skip_cluster_reset=True,load_docs_using=sirius_go_sdk --launch_sirius_docker --sirius_url http://localhost:49152 rebalance_in_with_compaction_and_ops (rebalance_new.rebalance_in.RebalanceInTests) Rebalances nodes into a cluster while doing ... Filename: debug.conf Prefix: rebalance_new.rebalance_in.RebalanceInTests Global Test input params: {'cluster_name': 'node', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'ini': 'node.ini', 'load_docs_using': 'sirius_go_sdk', 'num_nodes': 4, 'skip_cluster_reset': 'True', 'spec': 'debug'} Logs will be stored at /data/workspace/temp_centos7_4node_jython/logs/testrunner-24-Apr-24_03-32-12/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i node.ini -p get-cbcollect-info=False,skip_cluster_reset=True,load_docs_using=sirius_go_sdk -t rebalance_new.rebalance_in.RebalanceInTests.rebalance_in_with_compaction_and_ops,nodes_init=3,nodes_in=1,replicas=1,num_items=100000,doc_ops=create:update:delete' Test Input params: {'nodes_init': '3', 'nodes_in': '1', 'replicas': '1', 'num_items': '100000', 'doc_ops': 'create:update:delete', 'sirius_url': 'http://localhost:49152', 'ini': 'node.ini', 'cluster_name': 'node', 'spec': 'debug', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'skip_cluster_reset': 'True', 'load_docs_using': 'sirius_go_sdk', 'num_nodes': 4, 'no_of_test_identified': 1, 'case_number': 1, 'logs_folder': '/data/workspace/temp_centos7_4node_jython/logs/testrunner-24-Apr-24_03-32-12/test_1', 'rerun': False} 2024-04-24 03:32:15,712 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:32:15,712 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:32:15,714 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:32:15,714 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 03:32:15,809 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 03:32:15,810 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:32:15,816 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:32:15,817 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:32:15,973 | shell_util | INFO | ThreadPool_3 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:32:15,980 | shell_util | INFO | ThreadPool_2 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:32:15,981 | shell_util | INFO | ThreadPool_0 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:32:16,056 | shell_util | INFO | ThreadPool_1 | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:32:16,276 | shell_util | INFO | ThreadPool_3 | [shell_conn:extract_remote_info:521] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:32:16,277 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 03:32:16,283 | shell_util | INFO | ThreadPool_2 | [shell_conn:extract_remote_info:521] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:32:16,285 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:32:16,340 | shell_util | INFO | ThreadPool_0 | [shell_conn:extract_remote_info:521] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:32:16,341 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:32:16,360 | shell_util | INFO | ThreadPool_1 | [shell_conn:extract_remote_info:521] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:32:16,361 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:32:16,362 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 03:32:16,373 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:32:16,423 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:32:16,464 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:32:16,465 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= RebalanceInTests setup started for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 03:32:16,465 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:32:16,466 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:32:16,466 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:32:16,466 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 03:32:16,556 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:32:16,566 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:32:16,568 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:32:16,609 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 03:32:18,069 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:32:18,162 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:32:18,320 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:32:18,617 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:32:18,618 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:32:18,705 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:32:18,706 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= OnPremBaseTest setup started for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 03:32:18,736 | test | INFO | MainThread | [onPrem_basetestcase:setUp:270] Delete all buckets and rebalance out other nodes from 'C1' 2024-04-24 03:32:18,807 | test | WARNING | MainThread | [cluster_ready_functions:stop_running_rebalance:568] Rebalance stop was attempted but there is no ongoing rebalance 2024-04-24 03:32:32,560 | test | INFO | MainThread | [rebalance_util:monitor_rebalance:197] Rebalance done. Taken 11.128406047821045 secs to complete 2024-04-24 03:32:32,560 | test | INFO | MainThread | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait after rebalance complete 2024-04-24 03:32:42,930 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:32:43,028 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:32:45,028 | test | INFO | MainThread | [onPrem_basetestcase:enable_tls_on_nodes:535] Validating if services obey tls only on servers [ip:172.23.108.67 port:8091 ssh_username:root, ip:172.23.108.68 port:8091 ssh_username:root, ip:172.23.108.69 port:8091 ssh_username:root, ip:172.23.108.70 port:8091 ssh_username:root] 2024-04-24 03:32:45,029 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:32:45,120 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:32:45,509 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= OnPremBaseTest setup finished for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 03:32:45,553 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= OnPremBaseTest setup finished for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 03:33:00,793 | test | CRITICAL | ThreadPool_0 | [cluster_ready_functions:validate_orchestrator_selection:473] Orchestrator: 172.23.108.67 2024-04-24 03:33:00,933 | test | INFO | ThreadPool_0 | [table_view:display:72] Rebalance Overview +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 7.6.2-3551-enterprise / default | 3.049999997019768 | Cluster node | active / none | | 172.23.108.68 | None | kv | | | <--- IN --- | | | 172.23.108.69 | None | kv | | | <--- IN --- | | +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ 2024-04-24 03:33:01,043 | test | INFO | ThreadPool_0 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-04-24 03:33:16,149 | test | INFO | ThreadPool_0 | [task:check:1002] Rebalance - status: none, progress: 100 2024-04-24 03:33:16,213 | test | ERROR | ThreadPool_0 | [cluster_ready_functions:is_cluster_mixed:2263] Error while processing cluster info {'name': 'default', 'nodes': [{'clusterMembership': 'active', 'recoveryType': 'none', 'status': 'healthy', 'otpNode': 'ns_1@172.23.108.67', 'thisNode': True, 'hostname': '172.23.108.67:8091', 'nodeUUID': 'b0dfa8229d7dc061531ce7d61d12c39a', 'clusterCompatibility': 458758, 'version': '7.6.2-3551-enterprise', 'os': 'x86_64-pc-linux-gnu', 'cpuCount': 4, 'ports': {'direct': 11210, 'httpsMgmt': 18091, 'httpsCAPI': 18092, 'distTCP': 21100, 'distTLS': 21150}, 'services': ['kv'], 'nodeEncryption': True, 'nodeEncryptionClientCertVerification': False, 'addressFamilyOnly': False, 'configuredHostname': '172.23.108.67:8091', 'addressFamily': 'inet', 'externalListeners': [{'afamily': 'inet', 'nodeEncryption': True}], 'serverGroup': 'Group 1', 'couchApiBase': 'http://172.23.108.67:8092/', 'couchApiBaseHTTPS': 'https://172.23.108.67:18092/', 'nodeHash': 55023434, 'systemStats': {'cpu_utilization_rate': 6.149999983608723, 'cpu_stolen_rate': 0.2499999999417924, 'swap_total': 4550815744, 'swap_used': 24387584, 'mem_total': 4326473728, 'mem_free': 3086241792, 'mem_limit': 4326473728, 'cpu_cores_available': 4, 'allocstall': 823679}, 'interestingStats': {}, 'uptime': '713265', 'memoryTotal': 4326473728, 'memoryFree': 3086241792, 'mcdMemoryReserved': 3300, 'mcdMemoryAllocated': 3300}, {'clusterMembership': 'active', 'recoveryType': 'none', 'status': 'healthy', 'otpNode': 'ns_1@172.23.108.68', 'hostname': '172.23.108.68:8091', 'nodeUUID': '7a5407c8d68a11d844c353e59b75a3e1', 'clusterCompatibility': 458758, 'version': '7.6.2-3551-enterprise', 'os': 'x86_64-pc-linux-gnu', 'cpuCount': 4, 'ports': {'direct': 11210, 'httpsMgmt': 18091, 'httpsCAPI': 18092, 'distTCP': 21100, 'distTLS': 21150}, 'services': ['kv'], 'nodeEncryption': True, 'nodeEncryptionClientCertVerification': False, 'addressFamilyOnly': False, 'configuredHostname': '172.23.108.68:8091', 'addressFamily': 'inet', 'externalListeners': [{'afamily': 'inet', 'nodeEncryption': True}], 'serverGroup': 'Group 1', 'couchApiBase': 'http://172.23.108.68:8092/', 'couchApiBaseHTTPS': 'https://172.23.108.68:18092/', 'nodeHash': 85298659, 'systemStats': {'cpu_utilization_rate': 10.77857785182472, 'cpu_stolen_rate': 0.2500250025148034, 'swap_total': 4550815744, 'swap_used': 14417920, 'mem_total': 4326473728, 'mem_free': 3370733568, 'mem_limit': 4326473728, 'cpu_cores_available': 4, 'allocstall': 666}, 'interestingStats': {}, 'uptime': '713262', 'memoryTotal': 4326473728, 'memoryFree': 3370733568, 'mcdMemoryReserved': 3300, 'mcdMemoryAllocated': 3300}, {'clusterMembership': 'active', 'recoveryType': 'none', 'status': 'healthy', 'otpNode': 'ns_1@172.23.108.69', 'hostname': '172.23.108.69:8091', 'nodeUUID': '7decd50cebf6ec94b0d4442187ea147e', 'clusterCompatibility': 458758, 'version': '7.6.2-3551-enterprise', 'os': 'x86_64-pc-linux-gnu', 'cpuCount': 4, 'ports': {'direct': 11210, 'httpsMgmt': 18091, 'httpsCAPI': 18092, 'distTCP': 21100, 'distTLS': 21150}, 'services': ['kv'], 'nodeEncryption': True, 'nodeEncryptionClientCertVerification': False, 'addressFamilyOnly': False, 'configuredHostname': '172.23.108.69:8091', 'addressFamily': 'inet', 'externalListeners': [{'afamily': 'inet', 'nodeEncryption': True}], 'serverGroup': 'Group 1', 'couchApiBase': 'http://172.23.108.69:8092/', 'couchApiBaseHTTPS': 'https://172.23.108.69:18092/', 'nodeHash': 8143338, 'systemStats': {'cpu_utilization_rate': 2.074999995529652, 'cpu_stolen_rate': 0.2250000000640284, 'swap_total': 4550815744, 'swap_used': 16515072, 'mem_total': 4326473728, 'mem_free': 3362074624, 'mem_limit': 4326473728, 'cpu_cores_available': 4, 'allocstall': 161}, 'interestingStats': {}, 'uptime': '434085', 'memoryTotal': 4326473728, 'memoryFree': 3362074624, 'mcdMemoryReserved': 3300, 'mcdMemoryAllocated': 3300}], 'buckets': {'uri': '/pools/default/buckets?v=109581657&uuid=d9738d61943e243187cb367d88287fac', 'terseBucketsBase': '/pools/default/b/', 'terseStreamingBucketsBase': '/pools/default/bs/'}, 'bucketNames': [], 'remoteClusters': {'uri': '/pools/default/remoteClusters?uuid=d9738d61943e243187cb367d88287fac', 'validateURI': '/pools/default/remoteClusters?just_validate=1'}, 'alerts': [], 'alertsSilenceURL': '/controller/resetAlerts?uuid=d9738d61943e243187cb367d88287fac&token=0', 'controllers': {'addNode': {'uri': '/controller/addNodeV2?uuid=d9738d61943e243187cb367d88287fac'}, 'rebalance': {'uri': '/controller/rebalance?uuid=d9738d61943e243187cb367d88287fac'}, 'failOver': {'uri': '/controller/failOver?uuid=d9738d61943e243187cb367d88287fac'}, 'startGracefulFailover': {'uri': '/controller/startGracefulFailover?uuid=d9738d61943e243187cb367d88287fac'}, 'reAddNode': {'uri': '/controller/reAddNode?uuid=d9738d61943e243187cb367d88287fac'}, 'reFailOver': {'uri': '/controller/reFailOver?uuid=d9738d61943e243187cb367d88287fac'}, 'ejectNode': {'uri': '/controller/ejectNode?uuid=d9738d61943e243187cb367d88287fac'}, 'setRecoveryType': {'uri': '/controller/setRecoveryType?uuid=d9738d61943e243187cb367d88287fac'}, 'setAutoCompaction': {'uri': '/controller/setAutoCompaction?uuid=d9738d61943e243187cb367d88287fac', 'validateURI': '/controller/setAutoCompaction?just_validate=1'}, 'clusterLogsCollection': {'startURI': '/controller/startLogsCollection?uuid=d9738d61943e243187cb367d88287fac', 'cancelURI': '/controller/cancelLogsCollection?uuid=d9738d61943e243187cb367d88287fac'}, 'replication': {'createURI': '/controller/createReplication?uuid=d9738d61943e243187cb367d88287fac', 'validateURI': '/controller/createReplication?just_validate=1'}}, 'rebalanceStatus': 'none', 'rebalanceProgressUri': '/pools/default/rebalanceProgress', 'stopRebalanceUri': '/controller/stopRebalance?uuid=d9738d61943e243187cb367d88287fac', 'nodeStatusesUri': '/nodeStatuses', 'nodeServicesUri': '/pools/default/nodeServices?v=32181113', 'maxBucketCount': 30, 'maxCollectionCount': 1200, 'maxScopeCount': 1200, 'minReplicasCount': 0, 'autoCompactionSettings': {'parallelDBAndViewCompaction': False, 'magmaFragmentationPercentage': 50, 'databaseFragmentationThreshold': {'percentage': 30, 'size': 'undefined'}, 'viewFragmentationThreshold': {'percentage': 30, 'size': 'undefined'}, 'indexCompactionMode': 'circular', 'indexCircularCompaction': {'daysOfWeek': 'Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday', 'interval': {'fromHour': 0, 'toHour': 0, 'fromMinute': 0, 'toMinute': 0, 'abortOutside': False}}, 'indexFragmentationThreshold': {'percentage': 30}}, 'tasks': {'uri': '/pools/default/tasks?v=6501753'}, 'counters': {'rebalance_success': 64, 'rebalance_start': 64}, 'indexStatusURI': '/indexStatus?v=21137658', 'trustedCAsURI': '/pools/default/trustedCAs?v=36585939', 'clusterName': '', 'clusterEncryptionLevel': 'all', 'balanced': True, 'checkPermissionsURI': '/pools/default/checkPermissions?v=8uQWa7VuwFWDWIe%2FHPkMFNHKSUk%3D', 'memoryQuota': 2200, 'queryMemoryQuota': 0, 'indexMemoryQuota': 512, 'ftsMemoryQuota': 436, 'cbasMemoryQuota': 1239, 'eventingMemoryQuota': 256, 'storageTotals': {'ram': {'total': 12979421184, 'quotaTotal': 6920601600, 'quotaUsed': 0, 'used': 11177271296, 'usedByData': 0, 'quotaUsedPerNode': 0, 'quotaTotalPerNode': 2306867200}, 'hdd': {'total': 96139296768, 'quotaTotal': 96139296768, 'used': 16343680449, 'usedByData': 0, 'free': 79795616319}}, 'serverGroupsUri': '/pools/default/serverGroups?v=114066866'} 2024-04-24 03:33:16,213 | test | INFO | ThreadPool_0 | [task:check:1070] Rebalance completed with progress: 100% in 15.280362844467163 sec 2024-04-24 03:33:16,365 | test | CRITICAL | ThreadPool_0 | [cluster_ready_functions:validate_orchestrator_selection:473] Orchestrator: 172.23.108.67 2024-04-24 03:33:16,468 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:743] Cluster nodes..: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091'] 2024-04-24 03:33:16,469 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:743] KV............: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091'] 2024-04-24 03:33:16,585 | rest_api | INFO | ThreadPool_2 | [BucketOperations_Rest:create_bucket:390] Creating 'couchbase' bucket default 2024-04-24 03:33:16,647 | rest_api | DEBUG | ThreadPool_2 | [BucketOperations_Rest:create_bucket:399] 0.06 seconds to create bucket default 2024-04-24 03:33:16,675 | test | INFO | MainThread | [common_lib:sleep:21] Sleep 2 seconds. Reason: No servers_list found, Will retry... 2024-04-24 03:33:20,718 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:33:20,812 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:33:21,392 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:33:21,490 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:33:21,645 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:33:21,974 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:33:21,974 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:33:22,067 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:33:22,654 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:33:22,754 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:33:22,904 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:33:23,204 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:33:23,204 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:33:23,298 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/bulk-create /bulk-create {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 0, 'end': 100000, 'count': 100000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'insertOptions': {'expiry': 0, 'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 30}} http://localhost:49152/result /result {'seed': '1713954804113711487', 'deleteRecord': False} http://localhost:49152/result /result {'seed': '1713954804113711487', 'deleteRecord': False} 2024-04-24 03:33:36,389 | test | INFO | ThreadPool_0 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****.............................**...................................*....................................***X | 9389.100 | +-------+-----------------------------------------------------------------------------------------------------------------+----------+ 2024-04-24 03:33:36,390 | test | ERROR | ThreadPool_1 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' http://localhost:49152/retry-exceptions /retry-exceptions {'identifierToken': 'UNIQUE_1713954732', 'resultSeed': '1713954804113711487', 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 1}} http://localhost:49152/result /result {'seed': '1713954804113711487', 'deleteRecord': False} http://localhost:49152/result /result {'seed': '1713954804113711487', 'deleteRecord': False} 2024-04-24 03:34:36,418 | test | INFO | MainThread | [rebalance_base:setUp:153] Verifying num_items counts after doc_ops 2024-04-24 03:34:36,459 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:34:36,557 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:34:37,466 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:34:37,637 | shell_util | INFO | ThreadPool_0 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:34:38,462 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:34:38,567 | shell_util | INFO | ThreadPool_3 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:34:39,557 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:5751] Validating collection stats and item counts 2024-04-24 03:34:39,626 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:34:39,722 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:34:40,109 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:34:40,204 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:34:40,589 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:34:40,688 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:34:41,055 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:34:41,055 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:34:41,153 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:34:41,154 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:34:41,178 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:34:41,178 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:34:41,245 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:34:41,245 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:34:41,277 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:34:41,277 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:34:41,344 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:34:41,372 | shell_util | INFO | ThreadPool_2 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:34:41,928 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:34:42,027 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:34:42,027 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:34:42,129 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:34:42,129 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:34:42,228 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:34:42,670 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:34:42,766 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:34:42,766 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:34:42,865 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:34:42,865 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:34:42,964 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:34:43,153 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | 172.23.108.67 | Group 1 | kv | 4.1000 | 4.03 GiB | 2.78 GiB | 23.26 MiB / 4.24 GiB | 33400 / 33260 | 7.6.2-3551-enterprise / default | | 172.23.108.68 | Group 1 | kv | 3.8000 | 4.03 GiB | 2.99 GiB | 13.75 MiB / 4.24 GiB | 33317 / 33414 | 7.6.2-3551-enterprise / default | | 172.23.108.69 | Group 1 | kv | 3.8749 | 4.03 GiB | 2.99 GiB | 15.75 MiB / 4.24 GiB | 33283 / 33326 | 7.6.2-3551-enterprise / default | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ 2024-04-24 03:34:43,342 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+-----+ | Bucket | Type / Storage | Replicas | Rank | Vbuckets | Durability | TTL | Items | RAM Quota / Used | Disk Used | ARR | +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+-----+ | default | couchbase / magma | 1 | None | 1024 | none | 0 | 100000 | 768.00 MiB / 320.98 MiB | 251.60 MiB | 100 | +---------+-------------------+----------+------+----------+------------+-----+--------+-------------------------+------------+-----+ 2024-04-24 03:34:43,343 | test | INFO | MainThread | [cb_basetest:log_setup_status:211] ========= RebalanceBase setup complete for test #1 rebalance_in_with_compaction_and_ops ========= 2024-04-24 03:34:43,538 | test | INFO | MainThread | [common_lib:sleep:21] Sleep 10 seconds. Reason: wait for rebalance to start 2024-04-24 03:34:51,272 | test | CRITICAL | ThreadPool_3 | [cluster_ready_functions:validate_orchestrator_selection:473] Orchestrator: 172.23.108.67 2024-04-24 03:34:51,419 | test | INFO | ThreadPool_3 | [table_view:display:72] Rebalance Overview +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 7.6.2-3551-enterprise / default | 4.100000001490116 | Cluster node | active / none | | 172.23.108.68 | Group 1 | kv | 7.6.2-3551-enterprise / default | 3.800000008195639 | Cluster node | active / none | | 172.23.108.69 | Group 1 | kv | 7.6.2-3551-enterprise / default | 3.874999992549419 | Cluster node | active / none | | 172.23.108.70 | None | kv | | | <--- IN --- | | +---------------+---------+----------+---------------------------------+-------------------+--------------+-----------------------+ 2024-04-24 03:34:51,521 | test | INFO | ThreadPool_3 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-04-24 03:34:53,611 | rest_api | DEBUG | ThreadPool_1 | [BucketOperations_Rest:compact_bucket:647] Triggering bucket compaction for 'default' http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/bulk-upsert /bulk-upsert {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 0, 'end': 50000, 'count': 50000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'insertOptions': {'expiry': 0, 'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 5}} http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/result /result {'seed': '1713954893636687487', 'deleteRecord': False} 2024-04-24 03:34:53,644 | rest_api | DEBUG | ThreadPool_1 | [BucketOperations_Rest:compact_bucket:651] Bucket compaction successful http://localhost:49152/warmup-bucket /warmup-bucket {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}} http://localhost:49152/bulk-create /bulk-create {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 100000, 'end': 200000, 'count': 100000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'insertOptions': {'expiry': 0, 'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 5}} http://localhost:49152/bulk-delete /bulk-delete {'identifierToken': 'UNIQUE_1713954732', 'bucket': 'default', 'scope': '_default', 'collection': '_default', 'clusterConfig': {'username': 'Administrator', 'password': 'password', 'connectionString': 'couchbases://172.23.108.67', 'compressionConfig': {}, 'timeoutsConfig': {'connectTimeout': 20, 'KVTimeout': 10}}, 'operationConfig': {'docSize': 179, 'keyPrefix': 'test_docs', 'keySuffix': '', 'template': 'Person', 'start': 50000, 'end': 100000, 'count': 50000, 'docType': 'json', 'keySize': 0, 'readYourOwnWrite': False, 'fieldsToChange': [], 'exceptions': {'retryExceptions': [], 'ignoreExceptions': [], 'retryAttempts': 0}}, 'removeOptions': {'persistTo': 0, 'replicateTo': 0, 'durability': 'NONE', 'timeout': 5}} http://localhost:49152/result /result {'seed': '1713954893677835060', 'deleteRecord': False} http://localhost:49152/result /result {'seed': '1713954893697931688', 'deleteRecord': False} 2024-04-24 03:35:01,605 | test | ERROR | ThreadPool_3 | [task:call:829] 'ClusterUtils' object has no attribute 'test_log' 2024-04-24 03:35:01,670 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | Nodes | Zone | Services | CPU | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version / Config | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ | 172.23.108.67 | Group 1 | kv | 10.299 | 4.03 GiB | 2.75 GiB | 23.26 MiB / 4.24 GiB | 33052 / 33056 | 7.6.2-3551-enterprise / default | | 172.23.108.68 | Group 1 | kv | 6.8500 | 4.03 GiB | 2.97 GiB | 13.75 MiB / 4.24 GiB | 33317 / 33414 | 7.6.2-3551-enterprise / default | | 172.23.108.69 | Group 1 | kv | 7.3750 | 4.03 GiB | 2.97 GiB | 15.75 MiB / 4.24 GiB | 33283 / 33326 | 7.6.2-3551-enterprise / default | | 172.23.108.70 | Group 1 | kv | 2.0250 | 4.03 GiB | 3.15 GiB | 20.75 MiB / 4.24 GiB | 566 / 831 | 7.6.2-3551-enterprise / default | +---------------+---------+----------+--------+-----------+----------+----------------------+-------------------+---------------------------------+ 2024-04-24 03:35:01,670 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:35:01,827 | shell_util | INFO | ThreadPool_1 | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:35:04,278 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.67 - Parsing logs for error/critical string patterns 2024-04-24 03:35:04,278 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.67 with username:root, attempt#1 of 5 2024-04-24 03:35:04,352 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.67 as root 2024-04-24 03:35:04,508 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:35:04,803 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 http://localhost:49152/result /result {'seed': '1713954893636687487', 'deleteRecord': False} 2024-04-24 03:35:05,656 | test | ERROR | ThreadPool_0 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' http://localhost:49152/result /result {'seed': '1713954893677835060', 'deleteRecord': False} http://localhost:49152/result /result {'seed': '1713954893697931688', 'deleteRecord': False} 2024-04-24 03:35:05,714 | test | ERROR | ThreadPool_7 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' 2024-04-24 03:35:09,525 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 03:35:10,077 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... http://localhost:49152/result /result {'seed': '1713954893677835060', 'deleteRecord': False} 2024-04-24 03:35:18,393 | test | INFO | ThreadPool_4 | [table_view:display:72] Ops trend for bucket 'default' +----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****.........................**........................................................................******..*.*X | 15568.000 | | 6456.500 | ........................................*****X | 6456.500 | +----------+---------------------------------------------------------------------------------------------------------------------+-----------+ 2024-04-24 03:35:18,394 | test | ERROR | ThreadPool_5 | [task:call:8625] 'RestBasedDocLoader' object has no attribute '_condition' 2024-04-24 03:35:19,412 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.68 - Parsing logs for error/critical string patterns 2024-04-24 03:35:19,413 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.68 with username:root, attempt#1 of 5 2024-04-24 03:35:19,499 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.68 as root 2024-04-24 03:35:19,649 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:35:19,941 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:35:25,281 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 03:35:25,844 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-04-24 03:35:34,890 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.69 - Parsing logs for error/critical string patterns 2024-04-24 03:35:34,891 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.69 with username:root, attempt#1 of 5 2024-04-24 03:35:34,989 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.69 as root 2024-04-24 03:35:35,150 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:35:35,460 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:35:38,692 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 03:35:39,259 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-04-24 03:35:48,440 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:852] 172.23.108.70 - Parsing logs for error/critical string patterns 2024-04-24 03:35:48,440 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 03:35:48,531 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 03:35:48,684 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:35:48,988 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:35:48,989 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:128] SSH Connecting to 172.23.108.70 with username:root, attempt#1 of 5 2024-04-24 03:35:49,082 | shell_util | INFO | MainThread | [shell_conn:ssh_connect_with_retries:165] SSH Connected to 172.23.108.70 as root 2024-04-24 03:35:49,233 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:365] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-04-24 03:35:49,530 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:521] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-04-24 03:35:51,895 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-04-24 03:35:52,467 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-04-24 03:36:01,033 | test | CRITICAL | MainThread | [onPrem_basetestcase:tearDownEverything:661] Skipping get_trace !! FAIL ====================================================================== FAIL: rebalance_in_with_compaction_and_ops (rebalance_new.rebalance_in.RebalanceInTests) Rebalances nodes into a cluster while doing ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/workspace/temp_centos7_4node_jython/pytests/rebalance_new/rebalance_in.py", line 510, in rebalance_in_with_compaction_and_ops self.fail("Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 235.366s FAILED (failures=1) During the test, Remote Connections: 0, Disconnections: 0 SDK Connections: 0, Disconnections: 0 summary so far suite rebalance_new.rebalance_in.RebalanceInTests , pass 0 , fail 1 failures so far... rebalance_new.rebalance_in.RebalanceInTests.rebalance_in_with_compaction_and_ops testrunner logs, diags and results are available under /data/workspace/temp_centos7_4node_jython/logs/testrunner-24-Apr-24_03-32-12/test_1 + exit 0 Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Could not determine description. [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Sending email to: ashwin.govindarajulu@couchbase.com Finished: UNSTABLE