Started by user Ashwin Govindarajulu Rebuilds build #270 Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on slv-s83002 (deb12_jython_slave) in workspace /data/workspace/temp_job_ashwin The recommended git tool is: NONE using credential couchbaseqe-github Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/temp_job_ashwin # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > git --version # 'git version 2.39.2' using GIT_ASKPASS to set credentials Couchbase QE github account > /usr/bin/git fetch --tags --force --progress -- https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch > /usr/bin/git rev-parse origin/master_py3_dev^{commit} # timeout=10 Checking out Revision 2eebbe53dfb7a09f15a3bbb1d78f654f8892c5e0 (origin/master_py3_dev) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 2eebbe53dfb7a09f15a3bbb1d78f654f8892c5e0 # timeout=10 Commit message: "CBQE-8226: Adding ability to stop running task in e2e_app" > /usr/bin/git rev-list --no-walk 2eebbe53dfb7a09f15a3bbb1d78f654f8892c5e0 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #271 jenkins-temp_job_ashwin-271 # timeout=10 > /usr/bin/git remote # timeout=10 > /usr/bin/git submodule init # timeout=10 > /usr/bin/git submodule sync # timeout=10 > /usr/bin/git config --get remote.origin.url # timeout=10 Using shallow submodule update with depth 2 > /usr/bin/git submodule init # timeout=10 > /usr/bin/git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10 > /usr/bin/git config --get submodule.lib/capellaAPI.url # timeout=10 > /usr/bin/git remote # timeout=10 > /usr/bin/git config --get remote.origin.url # timeout=10 > /usr/bin/git config -f .gitmodules --get submodule.lib/capellaAPI.path # timeout=10 > /usr/bin/git config --get submodule.platform_utils/ssh_util.url # timeout=10 > /usr/bin/git remote # timeout=10 > /usr/bin/git config --get remote.origin.url # timeout=10 > /usr/bin/git config -f .gitmodules --get submodule.platform_utils/ssh_util.path # timeout=10 > /usr/bin/git config --get submodule.couchbase_utils/cb_server_rest_util.url # timeout=10 > /usr/bin/git remote # timeout=10 > /usr/bin/git config --get remote.origin.url # timeout=10 > /usr/bin/git config -f .gitmodules --get submodule.couchbase_utils/cb_server_rest_util.path # timeout=10 > /usr/bin/git config --get submodule.sirius.url # timeout=10 > /usr/bin/git remote # timeout=10 > /usr/bin/git config --get remote.origin.url # timeout=10 > /usr/bin/git config -f .gitmodules --get submodule.sirius.path # timeout=10 > /usr/bin/git config --get submodule.py_constants.url # timeout=10 > /usr/bin/git remote # timeout=10 > /usr/bin/git config --get remote.origin.url # timeout=10 > /usr/bin/git config -f .gitmodules --get submodule.py_constants.path # timeout=10 using GIT_ASKPASS to set credentials Couchbase QE github account > /usr/bin/git submodule update --init --recursive --remote --depth=2 lib/capellaAPI # timeout=10 using GIT_ASKPASS to set credentials Couchbase QE github account > /usr/bin/git submodule update --init --recursive --remote --depth=2 platform_utils/ssh_util # timeout=10 using GIT_ASKPASS to set credentials Couchbase QE github account > /usr/bin/git submodule update --init --recursive --remote --depth=2 couchbase_utils/cb_server_rest_util # timeout=10 using GIT_ASKPASS to set credentials Couchbase QE github account > /usr/bin/git submodule update --init --recursive --remote --depth=2 sirius # timeout=10 using GIT_ASKPASS to set credentials Couchbase QE github account > /usr/bin/git submodule update --init --recursive --remote --depth=2 py_constants # timeout=10 [temp_job_ashwin] $ /bin/bash -x /tmp/jenkins7877931390659223383.sh + ini_file=node.ini + echo ' [global] username:root password:couchbase port:8091 # data_path:/data [membase] rest_username:Administrator rest_password:password [servers] 1:172.23.108.67 2:172.23.108.68 3:172.23.108.69 4:172.23.108.70 5:172.23.108.72 6:172.23.108.74 ' + echo 'bucket_collections.travel_sample_app.TravelSampleApp: test_swap_services_between_nodes,cluster_conf=cluster,bucket_conf=travel_sample,rbac_conf=travel_sample_rbac,service_conf=services,use_https=True,enforce_tls=True,skip_setup_cleanup=True,initial_load=False ' + source /root/.bashrc ++ export PYENV_ROOT=/root/.pyenv ++ PYENV_ROOT=/root/.pyenv ++ export PATH=/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin ++ PATH=/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin ++ command -v pyenv +++ pyenv init - ++ eval 'PATH="$(bash --norc -ec '\''IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "'\'''\''/root/.pyenv/shims'\'''\''" ]]; then unset '\''\'\'''\''paths[i]'\''\'\'''\''; fi; done; echo "${paths[*]}"'\'')" export PATH="/root/.pyenv/shims:${PATH}" export PYENV_SHELL=bash source '\''/root/.pyenv/libexec/../completions/pyenv.bash'\'' command pyenv rehash 2>/dev/null pyenv() { local command command="${1:-}" if [ "$#" -gt 0 ]; then shift fi case "$command" in rehash|shell) eval "$(pyenv "sh-$command" "$@")" ;; *) command pyenv "$command" "$@" ;; esac }' ++++ bash --norc -ec 'IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "/root/.pyenv/shims" ]]; then unset '\''paths[i]'\''; fi; done; echo "${paths[*]}"' +++ PATH=/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin +++ export PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin +++ PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin +++ export PYENV_SHELL=bash +++ PYENV_SHELL=bash +++ source /root/.pyenv/libexec/../completions/pyenv.bash ++++ complete -F _pyenv pyenv +++ command pyenv rehash ++ export PYENV_ROOT=/root/.pyenv ++ PYENV_ROOT=/root/.pyenv ++ export PATH=/root/.pyenv/bin:/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin ++ PATH=/root/.pyenv/bin:/root/.pyenv/shims:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin ++ command -v pyenv +++ pyenv init - +++ local command +++ command=init +++ '[' 2 -gt 0 ']' +++ shift +++ case "$command" in +++ command pyenv init - ++ eval 'PATH="$(bash --norc -ec '\''IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "'\'''\''/root/.pyenv/shims'\'''\''" ]]; then unset '\''\'\'''\''paths[i]'\''\'\'''\''; fi; done; echo "${paths[*]}"'\'')" export PATH="/root/.pyenv/shims:${PATH}" export PYENV_SHELL=bash source '\''/root/.pyenv/libexec/../completions/pyenv.bash'\'' command pyenv rehash 2>/dev/null pyenv() { local command command="${1:-}" if [ "$#" -gt 0 ]; then shift fi case "$command" in rehash|shell) eval "$(pyenv "sh-$command" "$@")" ;; *) command pyenv "$command" "$@" ;; esac }' ++++ bash --norc -ec 'IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "/root/.pyenv/shims" ]]; then unset '\''paths[i]'\''; fi; done; echo "${paths[*]}"' +++ PATH=/root/.pyenv/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin +++ export PATH=/root/.pyenv/shims:/root/.pyenv/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin +++ PATH=/root/.pyenv/shims:/root/.pyenv/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin +++ export PYENV_SHELL=bash +++ PYENV_SHELL=bash +++ source /root/.pyenv/libexec/../completions/pyenv.bash ++++ complete -F _pyenv pyenv +++ command pyenv rehash + '[' master_py3_dev == master_py3_dev ']' + export PATH=/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin + PATH=/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin + go_version=1.22.4 + wget https://golang.org/dl/go1.22.4.linux-amd64.tar.gz --quiet + tar -xzf go1.22.4.linux-amd64.tar.gz + rm -f go1.22.4.linux-amd64.tar.gz ++ pwd + export GOPATH=/data/workspace/temp_job_ashwin/go + GOPATH=/data/workspace/temp_job_ashwin/go + export PATH=/data/workspace/temp_job_ashwin/go/bin:/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin + PATH=/data/workspace/temp_job_ashwin/go/bin:/usr/local/go/bin:/root/.pyenv/shims:/root/.pyenv/bin:/root/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin + export GO111MODULE=on + GO111MODULE=on + pyenv local 3.10.14 + local command + command=local + '[' 2 -gt 0 ']' + shift + case "$command" in + command pyenv local 3.10.14 + pip install -r requirements.txt Requirement already satisfied: beautifulsoup4 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (4.12.3) Requirement already satisfied: paramiko in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (3.4.0) Requirement already satisfied: requests in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (2.32.3) Requirement already satisfied: PyYAML in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 4)) (6.0.1) Requirement already satisfied: six in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (1.16.0) Requirement already satisfied: futures in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (3.0.5) Requirement already satisfied: certifi==2023.5.7 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (2023.5.7) Requirement already satisfied: couchbase==4.3.0 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (4.3.0) Requirement already satisfied: boto3 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (1.34.132) Requirement already satisfied: confluent-kafka in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (2.4.0) Requirement already satisfied: urllib3 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (2.2.2) Requirement already satisfied: setuptools in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (65.5.0) Requirement already satisfied: pytz in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (2024.1) Requirement already satisfied: soupsieve>1.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from beautifulsoup4->-r requirements.txt (line 1)) (2.5) Requirement already satisfied: pynacl>=1.5 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (1.5.0) Requirement already satisfied: cryptography>=3.3 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (42.0.8) Requirement already satisfied: bcrypt>=3.2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from paramiko->-r requirements.txt (line 2)) (4.1.3) Requirement already satisfied: idna<4,>=2.5 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (3.7) Requirement already satisfied: charset-normalizer<4,>=2 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from requests->-r requirements.txt (line 3)) (3.3.2) Requirement already satisfied: botocore<1.35.0,>=1.34.132 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from boto3->-r requirements.txt (line 9)) (1.34.132) Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from boto3->-r requirements.txt (line 9)) (1.0.1) Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from boto3->-r requirements.txt (line 9)) (0.10.2) Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from botocore<1.35.0,>=1.34.132->boto3->-r requirements.txt (line 9)) (2.9.0.post0) Requirement already satisfied: cffi>=1.12 in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from cryptography>=3.3->paramiko->-r requirements.txt (line 2)) (1.16.0) Requirement already satisfied: pycparser in /root/.pyenv/versions/3.10.14/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=3.3->paramiko->-r requirements.txt (line 2)) (2.22) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 24.2 [notice] To update, run: pip install --upgrade pip + '[' false == true ']' + port=49152 + INCR=1 ++ ss -tan4H 'sport = 49152' + '[' -n '' ']' + echo 49152 49152 + echo 'Will use 49152 for starting sirius' Will use 49152 for starting sirius + '[' 'git fetch https://review.couchbase.org/TAF refs/changes/67/213467/16 && git cherry-pick FETCH_HEAD' '!=' '' ']' + sh -c 'git fetch https://review.couchbase.org/TAF refs/changes/67/213467/16 && git cherry-pick FETCH_HEAD' From https://review.couchbase.org/TAF * branch refs/changes/67/213467/16 -> FETCH_HEAD [detached HEAD 6f07b1e9] CBQE-8226: Adding ability to reset cluster services to certain point Author: Ashwin Date: Thu Aug 8 11:30:13 2024 +0530 3 files changed, 217 insertions(+), 8 deletions(-) + python testrunner.py -c debug.conf -i node.ini -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,load_docs_using=sirius_go_sdk --launch_sirius_docker --sirius_url http://localhost:49152 test_swap_services_between_nodes (bucket_collections.travel_sample_app.TravelSampleApp) ... Filename: debug.conf Prefix: bucket_collections.travel_sample_app.TravelSampleApp Global Test input params: {'cluster_name': 'node', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'ini': 'node.ini', 'load_docs_using': 'sirius_go_sdk', 'num_nodes': 6, 'rerun': 'False', 'skip_cluster_reset': 'True', 'spec': 'debug'} Total test to be executed: 1 Logs will be stored at /data/workspace/temp_job_ashwin/logs/testrunner-24-Aug-08_23-16-44/test_1 python testrunner -i node.ini -p get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,load_docs_using=sirius_go_sdk -t bucket_collections.travel_sample_app.TravelSampleApp.test_swap_services_between_nodes,ini=node.ini,cluster_name=node,spec=debug,conf_file=debug.conf,get-cbcollect-info=False,skip_cluster_reset=True,rerun=False,load_docs_using=sirius_go_sdk,num_nodes=6,cluster_conf=cluster,bucket_conf=travel_sample,rbac_conf=travel_sample_rbac,service_conf=services,use_https=True,enforce_tls=True,skip_setup_cleanup=True,initial_load=False Test Input params: {'cluster_conf': 'cluster', 'bucket_conf': 'travel_sample', 'rbac_conf': 'travel_sample_rbac', 'service_conf': 'services', 'use_https': 'True', 'enforce_tls': 'True', 'skip_setup_cleanup': 'True', 'initial_load': 'False', 'sirius_url': 'http://localhost:49152', 'ini': 'node.ini', 'cluster_name': 'node', 'spec': 'debug', 'conf_file': 'debug.conf', 'get-cbcollect-info': 'False', 'skip_cluster_reset': 'True', 'rerun': 'False', 'load_docs_using': 'sirius_go_sdk', 'num_nodes': 6, 'no_of_test_identified': 1, 'case_number': 1, 'logs_folder': '/data/workspace/temp_job_ashwin/logs/testrunner-24-Aug-08_23-16-44/test_1'} 2024-08-08 23:16:55,016 | shell_util | INFO | ThreadPool_5 | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:55,151 | shell_util | INFO | ThreadPool_3 | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:55,188 | shell_util | INFO | ThreadPool_0 | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:55,189 | shell_util | INFO | ThreadPool_1 | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:55,190 | shell_util | INFO | ThreadPool_2 | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:55,197 | shell_util | INFO | ThreadPool_4 | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:55,278 | shell_util | INFO | ThreadPool_5 | [shell_conn:extract_remote_info:519] 172.23.108.74 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:16:55,459 | shell_util | INFO | ThreadPool_3 | [shell_conn:extract_remote_info:519] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:16:55,497 | shell_util | INFO | ThreadPool_0 | [shell_conn:extract_remote_info:519] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:16:55,500 | shell_util | INFO | ThreadPool_1 | [shell_conn:extract_remote_info:519] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:16:55,509 | shell_util | INFO | ThreadPool_4 | [shell_conn:extract_remote_info:519] 172.23.108.72 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:16:55,549 | shell_util | INFO | ThreadPool_2 | [shell_conn:extract_remote_info:519] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:16:55,645 | test | INFO | MainThread | [cb_basetest:log_setup_status:235] ========= TravelSampleApp setup started for test #1 test_swap_services_between_nodes ========= 2024-08-08 23:16:57,593 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:57,892 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:16:57,978 | test | INFO | MainThread | [cb_basetest:log_setup_status:235] ========= OnPremBaseTest setup started for test #1 test_swap_services_between_nodes ========= 2024-08-08 23:16:58,129 | test | INFO | MainThread | [cb_basetest:log_setup_status:235] ========= OnPremBaseTest setup finished for test #1 test_swap_services_between_nodes ========= 2024-08-08 23:16:58,129 | test | INFO | MainThread | [cb_basetest:log_setup_status:235] ========= AppBase setup started for test #1 test_swap_services_between_nodes ========= 2024-08-08 23:16:58,711 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:16:59,003 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:17:01,258 | test | INFO | MainThread | [app_basetest:__print_step:102] ############################################################ 1. Creating required travel-sample::SDK clients ############################################################ 2024-08-08 23:17:01,600 | test | INFO | MainThread | [app_basetest:__print_step:102] ############################################################ 2. Creating required couchstore::SDK clients ############################################################ 2024-08-08 23:17:01,631 | test | INFO | MainThread | [app_basetest:__print_step:102] ############################################################ 3. Creating required magma::SDK clients ############################################################ 2024-08-08 23:17:01,655 | test | INFO | MainThread | [app_basetest:__print_step:102] ############################################################ 4. Creating required ephemeral::SDK clients ############################################################ 2024-08-08 23:17:01,700 | test | INFO | MainThread | [cb_basetest:log_setup_status:235] ========= AppBase setup complete for test #1 test_swap_services_between_nodes ========= 2024-08-08 23:17:01,700 | test | INFO | MainThread | [cb_basetest:log_setup_status:235] ========= TravelSampleApp setup started for test #1 test_swap_services_between_nodes ========= 2024-08-08 23:17:01,707 | test | INFO | MainThread | [cb_basetest:log_setup_status:235] ========= TravelSampleApp setup complete for test #1 test_swap_services_between_nodes ========= 2024-08-08 23:17:01,725 | test | INFO | MainThread | [travel_sample_app:test_swap_services_between_nodes:276] Swap services between nodes 2024-08-08 23:17:02,433 | test | INFO | MainThread | [travel_sample_app:__set_nodes_with_desired_cluster_config:239] Reb topology: {'index': ['ns_1@172.23.108.69'], 'n1ql': ['ns_1@172.23.108.69'], 'eventing': ['ns_1@172.23.108.74'], 'cbas': ['ns_1@172.23.108.72'], 'backup': ['ns_1@172.23.108.70'], 'fts': ['ns_1@172.23.108.70']} 2024-08-08 23:17:02,632 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Londonderry, Total hits: with value 3, Time: elapsed: 0:00:00.224016, execution: 0:00:00.223944 2024-08-08 23:17:03,587 | test | INFO | ThreadPool_2 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Waiting before starting rebalance 2024-08-08 23:17:08,711 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 21 (from FRU), Query: IST -> FRU on days [2, 0, 1], time < AND s.utc > "22:30:00" AND s.utc <= "22:00:00">, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.997092, execution: 0:00:01.996974 2024-08-08 23:17:09,567 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Gloucestershire, WHERE (s.ratings.Overall) > 3 Total hits: with value 1, Time: elapsed: 0:00:00.285728, execution: 0:00:00.285668 2024-08-08 23:17:14,726 | test | CRITICAL | ThreadPool_2 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.70 2024-08-08 23:17:14,748 | test | INFO | Thread-67 | [user:run:303] agent_3 User - scenario_book_flight_with_return From TGU -> MIA for 3 people Booking success From MIA -> TGU for 4 people Return Booking success 2024-08-08 23:17:15,160 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 8 (from HNL), Query: MEL -> HNL on days [], time <>, stops <>, Total hits: with value 2, Time: elapsed: 0:00:02.329702, execution: 0:00:02.329637 2024-08-08 23:17:15,260 | test | INFO | ThreadPool_2 | [table_view:display:72] Rebalance Overview +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 8.0.0-1775-enterprise / default | 9.12499999627471 | Cluster node | active / none | | 172.23.108.68 | Group 1 | kv | 8.0.0-1775-enterprise / default | 9.524999996647239 | Cluster node | active / none | | 172.23.108.69 | Group 1 | cbas | 8.0.0-1775-enterprise / default | 3.449999997392297 | Cluster node | active / none | | 172.23.108.70 | Group 1 | eventing | 8.0.0-1775-enterprise / default | 3.625000007450581 | Cluster node | active / none | | 172.23.108.72 | Group 1 | index, n1ql | 8.0.0-1775-enterprise / default | 12.27500000037253 | Cluster node | active / none | | 172.23.108.74 | Group 1 | backup, fts | 8.0.0-1775-enterprise / default | 1.82499997317791 | Cluster node | active / none | +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ 2024-08-08 23:17:15,871 | test | INFO | ThreadPool_2 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-08-08 23:17:16,268 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Lancaster, WHERE (s.ratings.Overall) > 3 Total hits: with value 1, Time: elapsed: 0:00:00.406135, execution: 0:00:00.406045 2024-08-08 23:17:23,101 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 2 (from STT), Query: EIS -> STT on days [1, 6], time < AND s.utc <= "18:00:00">, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:02.171684, execution: 0:00:02.171615 Traceback (most recent call last): File "/data/workspace/temp_job_ashwin/pytests/bucket_collections/app/scenarios/user.py", line 300, in run self.result = User.scenarios[rand_scenario](self) File "/data/workspace/temp_job_ashwin/pytests/bucket_collections/app/scenarios/user.py", line 245, in scenario_book_one_way_flight b_summary = self.book_flight(u_id, self.scope) File "/data/workspace/temp_job_ashwin/pytests/bucket_collections/app/scenarios/user.py", line 126, in book_flight for row in result["q_result"].rows(): File "/root/.pyenv/versions/3.10.14/lib/python3.10/site-packages/couchbase/n1ql.py", line 96, in __next__ raise ex File "/root/.pyenv/versions/3.10.14/lib/python3.10/site-packages/couchbase/n1ql.py", line 90, in __next__ return self._get_next_row() File "/root/.pyenv/versions/3.10.14/lib/python3.10/site-packages/couchbase/n1ql.py", line 81, in _get_next_row raise ErrorMapper.build_exception(row) couchbase.exceptions.RequestCanceledException: RequestCanceledException() 2024-08-08 23:17:24,133 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Moss Beach, Total hits: with value 1, Time: elapsed: 0:00:00.234332, execution: 0:00:00.234264 2024-08-08 23:17:26,216 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:17:31,302 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 3 (from LHR), Query: EBB -> LHR on days [], time <>, stops < AND stops in [0]>, Total hits: with value 2, Time: elapsed: 0:00:02.215766, execution: 0:00:02.215695 2024-08-08 23:17:32,269 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Gillingham, WHERE (s.ratings.Overall) > 4 Total hits: with value 1, Time: elapsed: 0:00:00.219247, execution: 0:00:00.219097 2024-08-08 23:17:33,118 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Annan, Total hits: with value 2, Time: elapsed: 0:00:00.246382, execution: 0:00:00.246280 2024-08-08 23:17:36,775 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:17:37,991 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 9 (from LAS), Query: SGF -> LAS on days [6, 4, 2], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.848443, execution: 0:00:01.848328 2024-08-08 23:17:38,630 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::King City, WHERE (s.ratings.Overall) > 1 Total hits: with value 2, Time: elapsed: 0:00:00.224691, execution: 0:00:00.224559 2024-08-08 23:17:45,543 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 1 (from STN), Query: PMF -> STN on days [5, 1, 2, 6, 4], time < AND s.utc > "4:00:00" AND s.utc <= "9:30:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:02.835885, execution: 0:00:02.835819 2024-08-08 23:17:46,281 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Dundee, WHERE (s.ratings.Overall) > 3 Total hits: with value 1, Time: elapsed: 0:00:00.248716, execution: 0:00:00.248648 2024-08-08 23:17:47,416 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 35.6 2024-08-08 23:17:51,533 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 2 (from MSP), Query: INL -> MSP on days [4, 3, 0, 2, 5], time < AND s.utc <= "4:00:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.994182, execution: 0:00:01.994058 2024-08-08 23:17:52,323 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Vaison-la-Romaine, WHERE (s.ratings.Overall) > 4 Total hits: with value 4, Time: elapsed: 0:00:00.253816, execution: 0:00:00.253747 2024-08-08 23:17:58,138 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 37.96 2024-08-08 23:17:58,521 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 2 (from STN), Query: EFL -> STN on days [1, 6, 5, 0, 4], time < AND s.utc > "0:30:00" AND s.utc <= "14:00:00">, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:02.366017, execution: 0:00:02.365883 2024-08-08 23:18:03,079 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 12 (from ANC), Query: JNU -> ANC on days [0, 2, 3, 6], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.745287, execution: 0:00:01.745184 2024-08-08 23:18:08,767 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 38.32 2024-08-08 23:18:09,447 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 9 (from ATL), Query: ROA -> ATL on days [4], time <>, stops < AND stops in [0]>, Total hits: with value 4, Time: elapsed: 0:00:01.804563, execution: 0:00:01.804444 2024-08-08 23:18:10,030 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Vittel, Total hits: with value 1, Time: elapsed: 0:00:00.225149, execution: 0:00:00.225058 2024-08-08 23:18:15,861 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 1 (from YKS), Query: IKS -> YKS on days [4, 5, 6], time < AND s.utc > "8:00:00">, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.718591, execution: 0:00:01.718472 2024-08-08 23:18:16,713 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Helensburgh, Total hits: with value 3, Time: elapsed: 0:00:00.266634, execution: 0:00:00.266563 2024-08-08 23:18:19,714 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 39.03 2024-08-08 23:18:21,396 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 2 (from BET), Query: KKH -> BET on days [2, 6, 1, 0], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.732386, execution: 0:00:01.732137 2024-08-08 23:18:21,893 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Hay-on-Wye, WHERE (s.ratings.Overall) > 5 Total hits: with value 1, Time: elapsed: 0:00:00.158524, execution: 0:00:00.158463 2024-08-08 23:18:22,833 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Santa Barbara, Total hits: with value 1, Time: elapsed: 0:00:00.320788, execution: 0:00:00.320722 2024-08-08 23:18:23,702 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Rouen, WHERE (s.ratings.Overall) > 5 Total hits: with value 1, Time: elapsed: 0:00:00.217193, execution: 0:00:00.217124 2024-08-08 23:18:24,549 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Vittel, WHERE (s.ratings.Overall) > 3 Total hits: with value 1, Time: elapsed: 0:00:00.233910, execution: 0:00:00.233828 2024-08-08 23:18:30,194 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 4 (from MAD), Query: LDE -> MAD on days [4, 6, 5, 2, 3, 1], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:02.352870, execution: 0:00:02.352617 2024-08-08 23:18:30,383 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 39.39 2024-08-08 23:18:30,809 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Bishop, Total hits: with value 1, Time: elapsed: 0:00:00.181028, execution: 0:00:00.180947 2024-08-08 23:18:31,476 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Altrincham, Total hits: with value 1, Time: elapsed: 0:00:00.176676, execution: 0:00:00.176539 2024-08-08 23:18:36,502 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 2 (from HRB), Query: HEK -> HRB on days [1, 5, 4], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:02.490693, execution: 0:00:02.490441 2024-08-08 23:18:37,127 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Puteaux, WHERE (s.ratings.Overall) > 3 Total hits: with value 3, Time: elapsed: 0:00:00.212349, execution: 0:00:00.212281 2024-08-08 23:18:41,080 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 40.46 2024-08-08 23:18:42,436 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 2 (from YIW), Query: JJN -> YIW on days [], time < AND s.utc <= "11:00:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:02.544366, execution: 0:00:02.544271 2024-08-08 23:18:46,851 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 2 (from HAK), Query: TXN -> HAK on days [0], time < AND s.utc <= "13:00:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.940276, execution: 0:00:01.940138 2024-08-08 23:18:47,782 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Bakewell, Total hits: with value 1, Time: elapsed: 0:00:00.247941, execution: 0:00:00.247850 2024-08-08 23:18:51,950 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 40.81 2024-08-08 23:18:53,357 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 3 (from CDG), Query: MSQ -> CDG on days [3, 2, 6, 4, 1], time < AND s.utc > "5:00:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:02.130643, execution: 0:00:02.130502 2024-08-08 23:18:58,203 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 1 (from STN), Query: PEG -> STN on days [0, 6, 2, 1], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.790023, execution: 0:00:01.789785 2024-08-08 23:19:02,497 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 41.52 2024-08-08 23:19:04,364 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 2 (from AIN), Query: ATK -> AIN on days [], time <>, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.714446, execution: 0:00:01.714369 2024-08-08 23:19:04,968 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Los Angeles, Total hits: with value 35, Time: elapsed: 0:00:00.176441, execution: 0:00:00.176328 2024-08-08 23:19:05,523 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Argyll and Bute, WHERE (s.ratings.Overall) > 4 Total hits: with value 6, Time: elapsed: 0:00:00.180191, execution: 0:00:00.180109 2024-08-08 23:19:06,361 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Pwllheli, Total hits: with value 1, Time: elapsed: 0:00:00.229841, execution: 0:00:00.229775 2024-08-08 23:19:11,323 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 7 (from BRS), Query: POZ -> BRS on days [5, 3], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.733595, execution: 0:00:01.733491 2024-08-08 23:19:13,060 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 42.24 2024-08-08 23:19:16,360 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 2 (from BRW), Query: PIZ -> BRW on days [0, 2], time < AND s.utc > "12:00:00" AND s.utc <= "16:00:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.720630, execution: 0:00:01.720523 2024-08-08 23:19:22,639 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 14 (from PTP), Query: SDQ -> PTP on days [1, 6, 2, 0], time < AND s.utc <= "19:00:00">, stops < AND stops in [0]>, Total hits: with value 2, Time: elapsed: 0:00:01.762427, execution: 0:00:01.762299 2024-08-08 23:19:23,335 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Carmel Valley, WHERE (s.ratings.Overall) > 3 Total hits: with value 1, Time: elapsed: 0:00:00.171368, execution: 0:00:00.171225 2024-08-08 23:19:23,670 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 42.59 2024-08-08 23:19:28,439 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 2 (from ANC), Query: STG -> ANC on days [3, 0, 5, 4], time <>, stops <>, Total hits: with value 2, Time: elapsed: 0:00:01.743782, execution: 0:00:01.743694 2024-08-08 23:19:29,007 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::None, WHERE (s.ratings.Overall) > 2 Total hits: with value 0, Time: elapsed: 0:00:00.166347, execution: 0:00:00.166232 2024-08-08 23:19:29,639 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Pontardawe, WHERE (s.ratings.Overall) > 4 Total hits: with value 1, Time: elapsed: 0:00:00.196176, execution: 0:00:00.196029 2024-08-08 23:19:34,289 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 43.3 2024-08-08 23:19:35,955 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 1 (from ATL), Query: GTR -> ATL on days [], time <>, stops < AND stops in [0]>, Total hits: with value 3, Time: elapsed: 0:00:01.777021, execution: 0:00:01.776950 2024-08-08 23:19:36,559 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Carpentras, Total hits: with value 1, Time: elapsed: 0:00:00.164260, execution: 0:00:00.164154 2024-08-08 23:19:37,444 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Moray, WHERE (s.ratings.Overall) > 5 Total hits: with value 1, Time: elapsed: 0:00:00.323064, execution: 0:00:00.322923 2024-08-08 23:19:42,603 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 13 (from EWR), Query: YEG -> EWR on days [4, 3, 2, 6, 0], time <>, stops <>, Total hits: with value 2, Time: elapsed: 0:00:01.756747, execution: 0:00:01.756596 2024-08-08 23:19:43,299 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::West Hollywood, WHERE (s.ratings.Overall) > 1 Total hits: with value 1, Time: elapsed: 0:00:00.279692, execution: 0:00:00.279577 2024-08-08 23:19:44,889 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 43.66 2024-08-08 23:19:48,237 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 9 (from RTM), Query: EGC -> RTM on days [5, 1], time < AND s.utc > "3:30:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.746620, execution: 0:00:01.746400 2024-08-08 23:19:53,340 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 3 (from CDG), Query: CKY -> CDG on days [], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.793836, execution: 0:00:01.793703 2024-08-08 23:19:55,481 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 44.73 2024-08-08 23:19:59,672 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 12 (from ADL), Query: BNE -> ADL on days [4, 1, 6, 2, 0], time <>, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.698537, execution: 0:00:01.698433 2024-08-08 23:20:00,421 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Puy-l'Évêque, WHERE (s.ratings.Overall) > 2 Total hits: with value 1, Time: elapsed: 0:00:00.189068, execution: 0:00:00.188925 2024-08-08 23:20:00,968 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Half Moon Bay, WHERE (s.ratings.Overall) > 4 Total hits: with value 1, Time: elapsed: 0:00:00.176334, execution: 0:00:00.176059 2024-08-08 23:20:06,255 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 45.09 2024-08-08 23:20:06,957 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 2 (from KTN), Query: SIT -> KTN on days [3, 4, 5, 0], time <>, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.686272, execution: 0:00:01.686144 2024-08-08 23:20:07,609 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Barrowford, WHERE (s.ratings.Overall) > 4 Total hits: with value 1, Time: elapsed: 0:00:00.165469, execution: 0:00:00.165363 2024-08-08 23:20:13,985 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 8 (from FRU), Query: OSS -> FRU on days [5, 2, 6, 4], time < AND s.utc <= "0:00:00">, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.984969, execution: 0:00:01.984885 2024-08-08 23:20:16,860 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 45.8 2024-08-08 23:20:19,094 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 1 (from TLJ), Query: TCT -> TLJ on days [2, 1, 5, 0, 4, 3], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.853128, execution: 0:00:01.852942 2024-08-08 23:20:25,529 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 3 (from LAS), Query: CPR -> LAS on days [2, 1, 6, 5, 3], time < AND s.utc > "14:00:00" AND s.utc <= "17:30:00">, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.776375, execution: 0:00:01.776124 2024-08-08 23:20:27,816 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 46.51 2024-08-08 23:20:31,958 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 48 (from EGC), Query: LPL -> EGC on days [6, 3, 2, 1], time <>, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.875423, execution: 0:00:01.875290 2024-08-08 23:20:32,837 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Mariposa County, WHERE (s.ratings.Overall) > 2 Total hits: with value 9, Time: elapsed: 0:00:00.258602, execution: 0:00:00.258490 2024-08-08 23:20:33,563 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Saint-Marcellin-lès-Vaison, WHERE (s.ratings.Overall) > 5 Total hits: with value 1, Time: elapsed: 0:00:00.232045, execution: 0:00:00.231965 2024-08-08 23:20:34,315 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Hale Barns, WHERE (s.ratings.Overall) > 3 Total hits: with value 1, Time: elapsed: 0:00:00.221427, execution: 0:00:00.221288 2024-08-08 23:20:38,552 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 46.87 2024-08-08 23:20:40,606 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 2 (from ANU), Query: TAB -> ANU on days [0], time <>, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:02.402040, execution: 0:00:02.401928 2024-08-08 23:20:46,592 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 1 (from MRS), Query: AGB -> MRS on days [4, 3, 2, 5], time <>, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.870394, execution: 0:00:01.870296 2024-08-08 23:20:47,228 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Calistoga, Total hits: with value 2, Time: elapsed: 0:00:00.221146, execution: 0:00:00.221005 2024-08-08 23:20:47,817 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Groveland, Total hits: with value 1, Time: elapsed: 0:00:00.167401, execution: 0:00:00.167319 2024-08-08 23:20:49,267 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 44.78 2024-08-08 23:20:52,884 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 1 (from BOS), Query: RUT -> BOS on days [3, 2, 6, 5, 4, 1], time < AND s.utc > "4:00:00" AND s.utc <= "16:30:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.849709, execution: 0:00:01.849564 2024-08-08 23:20:58,564 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 2 (from ATL), Query: LWB -> ATL on days [2, 1, 3], time <>, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.870427, execution: 0:00:01.870319 2024-08-08 23:21:00,138 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: running, progress: 66.67 2024-08-08 23:21:04,154 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 14 (from LTN), Query: FUE -> LTN on days [], time < AND s.utc > "5:00:00">, stops < AND stops in [0]>, Total hits: with value 2, Time: elapsed: 0:00:01.794184, execution: 0:00:01.794085 2024-08-08 23:21:09,664 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_fights_with_stop_count Total src_airports: 1368 , Total dest_airports: 1 (from CAI), Query: TIF -> CAI on days [4, 5, 3, 1, 0, 6], time < AND s.utc <= "0:30:00">, stops < AND stops in [0]>, Total hits: with value 1, Time: elapsed: 0:00:01.769530, execution: 0:00:01.769425 2024-08-08 23:21:14,420 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 1 (from TLJ), Query: TCT -> TLJ on days [0, 2, 6, 4], time < AND s.utc <= "18:30:00">, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.760920, execution: 0:00:01.760796 2024-08-08 23:21:16,054 | test | INFO | ThreadPool_2 | [task:check:912] Rebalance - status: none, progress: 100 2024-08-08 23:21:16,333 | test | INFO | ThreadPool_2 | [task:check:980] Rebalance completed with progress: 100% in 241.07293152809143 sec 2024-08-08 23:21:16,851 | test | INFO | ThreadPool_2 | [common_lib:sleep:21] Sleep 2 seconds. Reason: orchestrator='undefined' 2024-08-08 23:21:18,767 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days_time Total src_airports: 1368 , Total dest_airports: 4 (from ATL), Query: FAY -> ATL on days [4, 5, 2, 0], time < AND s.utc <= "8:00:00">, stops <>, Total hits: with value 3, Time: elapsed: 0:00:01.686761, execution: 0:00:01.686700 2024-08-08 23:21:19,010 | test | INFO | ThreadPool_2 | [common_lib:sleep:21] Sleep 2 seconds. Reason: orchestrator='undefined' 2024-08-08 23:21:19,329 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Puy-l'Évêque, Total hits: with value 1, Time: elapsed: 0:00:00.160340, execution: 0:00:00.160277 2024-08-08 23:21:19,890 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Birkenhead, WHERE (s.ratings.Overall) > 3 Total hits: with value 2, Time: elapsed: 0:00:00.188618, execution: 0:00:00.188484 2024-08-08 23:21:20,554 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_search_available_hotels Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Derbyshire, Total hits: with value 8, Time: elapsed: 0:00:00.200848, execution: 0:00:00.200691 2024-08-08 23:21:21,105 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Criccieth, WHERE (s.ratings.Overall) > 2 Total hits: with value 1, Time: elapsed: 0:00:00.146223, execution: 0:00:00.146143 2024-08-08 23:21:21,130 | test | INFO | ThreadPool_2 | [common_lib:sleep:21] Sleep 2 seconds. Reason: orchestrator='undefined' 2024-08-08 23:21:23,977 | test | CRITICAL | ThreadPool_2 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.74 2024-08-08 23:21:24,671 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] Cluster nodes..: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091', '172.23.108.70:18091', '172.23.108.72:18091', '172.23.108.74:18091'] 2024-08-08 23:21:24,677 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] KV............: ['172.23.108.67:18091', '172.23.108.68:18091'] 2024-08-08 23:21:24,692 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] Index.........: ['172.23.108.69:18091'] 2024-08-08 23:21:24,703 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] Query.........: ['172.23.108.69:18091'] 2024-08-08 23:21:24,714 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] CBAS..........: ['172.23.108.72:18091'] 2024-08-08 23:21:24,729 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] FTS...........: ['172.23.108.70:18091'] 2024-08-08 23:21:24,735 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] Eventing......: ['172.23.108.74:18091'] 2024-08-08 23:21:24,750 | infra | CRITICAL | ThreadPool_2 | [task:print_nodes:642] Backup........: ['172.23.108.70:18091'] 2024-08-08 23:21:25,038 | test | INFO | MainThread | [travel_sample_app:trigger_remap_service_rebalance:259] {'index': ['ns_1@172.23.108.72'], 'n1ql': ['ns_1@172.23.108.72'], 'eventing': ['ns_1@172.23.108.70'], 'cbas': ['ns_1@172.23.108.69'], 'backup': ['ns_1@172.23.108.74'], 'fts': ['ns_1@172.23.108.74']} 2024-08-08 23:21:25,645 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 14 (from PIE), Query: GSO -> PIE on days [2, 1, 6, 4, 0], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.996521, execution: 0:00:01.996402 2024-08-08 23:21:25,873 | test | INFO | ThreadPool_3 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Waiting before starting rebalance 2024-08-08 23:21:30,088 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 47 (from BWI), Query: BNA -> BWI on days [4], time <>, stops <>, Total hits: with value 2, Time: elapsed: 0:00:01.798484, execution: 0:00:01.798330 2024-08-08 23:21:34,546 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 2 (from CKG), Query: XNN -> CKG on days [6, 3], time <>, stops <>, Total hits: with value 1, Time: elapsed: 0:00:01.814632, execution: 0:00:01.814498 2024-08-08 23:21:35,120 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_read_hotel_reviews Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::Thousand Oaks, WHERE (s.ratings.Overall) > 4 Total hits: with value 1, Time: elapsed: 0:00:00.169556, execution: 0:00:00.169488 2024-08-08 23:21:35,761 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 59, Query: Hotels with avg_rating in United States::New Cuyama, WHERE (s.ratings.Overall) > 2 Total hits: with value 1, Time: elapsed: 0:00:00.194013, execution: 0:00:00.193931 2024-08-08 23:21:36,268 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 172, Query: Hotels with avg_rating in United Kingdom::Salford, WHERE (s.ratings.Overall) > 3 Total hits: with value 1, Time: elapsed: 0:00:00.162748, execution: 0:00:00.162675 2024-08-08 23:21:36,862 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_hotel_based_on_ratings Total Country: 3, Total Cities in selected country: 43, Query: Hotels with avg_rating in France::Paris-8E-Arrondissement, WHERE (s.ratings.Overall) > 2 Total hits: with value 3, Time: elapsed: 0:00:00.166070, execution: 0:00:00.165821 2024-08-08 23:21:36,972 | test | CRITICAL | ThreadPool_3 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.74 2024-08-08 23:21:37,491 | test | INFO | ThreadPool_3 | [table_view:display:72] Rebalance Overview +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 8.0.0-1775-enterprise / default | 37.27500000037253 | Cluster node | active / none | | 172.23.108.68 | Group 1 | kv | 8.0.0-1775-enterprise / default | 33.04999999701977 | Cluster node | active / none | | 172.23.108.69 | Group 1 | index, n1ql | 8.0.0-1775-enterprise / default | 83.79999999888241 | Cluster node | active / none | | 172.23.108.70 | Group 1 | backup, fts | 8.0.0-1775-enterprise / default | 5.175000000745058 | Cluster node | active / none | | 172.23.108.72 | Group 1 | cbas | 8.0.0-1775-enterprise / default | 33.10000000521541 | Cluster node | active / none | | 172.23.108.74 | Group 1 | eventing | 8.0.0-1775-enterprise / default | 3.075000047683716 | Cluster node | active / none | +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ 2024-08-08 23:21:38,006 | test | INFO | ThreadPool_3 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-08-08 23:21:41,579 | test | INFO | Thread-68 | [guest:run:120] agent_4 Guest - scenario_query_routes_on_days Total src_airports: 1368 , Total dest_airports: 187 (from BZE), Query: DFW -> BZE on days [], time <>, stops <>, Total hits: with value 2, Time: elapsed: 0:00:01.789666, execution: 0:00:01.789487 Traceback (most recent call last): File "/data/workspace/temp_job_ashwin/pytests/bucket_collections/app/scenarios/guest.py", line 116, in run self.result = Guest.scenarios[ File "/data/workspace/temp_job_ashwin/pytests/bucket_collections/app/scenarios/guest.py", line 71, in scenario_query_routes_on_days_time result = query_util.Airline.query_for_routes(sdk_clients["guest"], File "/data/workspace/temp_job_ashwin/pytests/bucket_collections/app/lib/query_util.py", line 82, in query_for_routes dest_airports = Airline.get_destination_airport_from_selected_src( File "/data/workspace/temp_job_ashwin/pytests/bucket_collections/app/lib/query_util.py", line 64, in get_destination_airport_from_selected_src for row in get_dest_airports.rows(): File "/root/.pyenv/versions/3.10.14/lib/python3.10/site-packages/couchbase/n1ql.py", line 96, in __next__ raise ex File "/root/.pyenv/versions/3.10.14/lib/python3.10/site-packages/couchbase/n1ql.py", line 90, in __next__ return self._get_next_row() File "/root/.pyenv/versions/3.10.14/lib/python3.10/site-packages/couchbase/n1ql.py", line 81, in _get_next_row raise ErrorMapper.build_exception(row) couchbase.exceptions.RequestCanceledException: RequestCanceledException() 2024-08-08 23:21:48,359 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:21:59,109 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:22:09,805 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:22:20,452 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 35.11 2024-08-08 23:22:31,038 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 35.47 2024-08-08 23:22:41,564 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 36.54 2024-08-08 23:22:52,040 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 36.89 2024-08-08 23:23:02,717 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 40.1 2024-08-08 23:23:13,346 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 41.16 2024-08-08 23:23:23,943 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 41.88 2024-08-08 23:23:34,580 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 42.59 2024-08-08 23:23:45,281 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 42.95 2024-08-08 23:23:56,069 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 43.66 2024-08-08 23:24:06,888 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 44.37 2024-08-08 23:24:17,458 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 45.09 2024-08-08 23:24:27,979 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 45.44 2024-08-08 23:24:38,611 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 46.15 2024-08-08 23:24:49,249 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 46.87 2024-08-08 23:24:59,868 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 44.44 2024-08-08 23:25:10,539 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: running, progress: 55.44 2024-08-08 23:25:26,477 | test | INFO | ThreadPool_3 | [task:check:912] Rebalance - status: none, progress: 100 2024-08-08 23:25:26,806 | test | INFO | ThreadPool_3 | [task:check:980] Rebalance completed with progress: 100% in 229.31618881225586 sec 2024-08-08 23:25:27,169 | test | INFO | ThreadPool_3 | [common_lib:sleep:21] Sleep 2 seconds. Reason: orchestrator='undefined' 2024-08-08 23:25:29,404 | test | INFO | ThreadPool_3 | [common_lib:sleep:21] Sleep 2 seconds. Reason: orchestrator='undefined' 2024-08-08 23:25:31,564 | test | INFO | ThreadPool_3 | [common_lib:sleep:21] Sleep 2 seconds. Reason: orchestrator='undefined' 2024-08-08 23:25:34,397 | test | CRITICAL | ThreadPool_3 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.70 2024-08-08 23:25:35,278 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] Cluster nodes..: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091', '172.23.108.70:18091', '172.23.108.72:18091', '172.23.108.74:18091'] 2024-08-08 23:25:35,278 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] KV............: ['172.23.108.67:18091', '172.23.108.68:18091'] 2024-08-08 23:25:35,278 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] Index.........: ['172.23.108.72:18091'] 2024-08-08 23:25:35,278 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] Query.........: ['172.23.108.72:18091'] 2024-08-08 23:25:35,278 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] CBAS..........: ['172.23.108.69:18091'] 2024-08-08 23:25:35,279 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] FTS...........: ['172.23.108.74:18091'] 2024-08-08 23:25:35,279 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] Eventing......: ['172.23.108.70:18091'] 2024-08-08 23:25:35,279 | infra | CRITICAL | ThreadPool_3 | [task:print_nodes:642] Backup........: ['172.23.108.74:18091'] 2024-08-08 23:25:35,284 | test | INFO | MainThread | [travel_sample_app:test_swap_services_between_nodes:291] Test of service-less node swap 2024-08-08 23:25:36,010 | test | INFO | MainThread | [travel_sample_app:__set_nodes_with_desired_cluster_config:239] Reb topology: {'index': ['ns_1@172.23.108.69'], 'n1ql': ['ns_1@172.23.108.69'], 'eventing': ['ns_1@172.23.108.72'], 'cbas': ['ns_1@172.23.108.72'], 'backup': ['ns_1@172.23.108.70'], 'fts': ['ns_1@172.23.108.70']} 2024-08-08 23:25:37,384 | test | INFO | ThreadPool_0 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Waiting before starting rebalance 2024-08-08 23:25:48,190 | test | CRITICAL | ThreadPool_0 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.70 2024-08-08 23:25:48,713 | test | INFO | ThreadPool_0 | [table_view:display:72] Rebalance Overview +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 8.0.0-1775-enterprise / default | 13.89999999664724 | Cluster node | active / none | | 172.23.108.68 | Group 1 | kv | 8.0.0-1775-enterprise / default | 11.37500000186265 | Cluster node | active / none | | 172.23.108.69 | Group 1 | cbas | 8.0.0-1775-enterprise / default | 37.39999999292195 | Cluster node | active / none | | 172.23.108.70 | Group 1 | eventing | 8.0.0-1775-enterprise / default | 8.650000002235174 | Cluster node | active / none | | 172.23.108.72 | Group 1 | index, n1ql | 8.0.0-1775-enterprise / default | 7.225000001490116 | Cluster node | active / none | | 172.23.108.74 | Group 1 | backup, fts | 8.0.0-1775-enterprise / default | 5.415541595882843 | Cluster node | active / none | +---------------+---------+-------------+---------------------------------+-------------------+--------------+-----------------------+ 2024-08-08 23:25:49,375 | test | INFO | ThreadPool_0 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-08-08 23:25:59,728 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:26:10,417 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:26:21,013 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 34.72 2024-08-08 23:26:31,658 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 35.11 2024-08-08 23:26:42,356 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 35.83 2024-08-08 23:26:52,950 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 36.54 2024-08-08 23:27:03,754 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 36.89 2024-08-08 23:27:14,719 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 37.61 2024-08-08 23:27:25,232 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 37.96 2024-08-08 23:27:35,506 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 41.88 2024-08-08 23:27:45,666 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 42.59 2024-08-08 23:27:55,832 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 42.95 2024-08-08 23:28:05,986 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 43.66 2024-08-08 23:28:16,130 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 44.02 2024-08-08 23:28:26,280 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 45.09 2024-08-08 23:28:36,447 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 45.44 2024-08-08 23:28:46,579 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 46.15 2024-08-08 23:28:56,720 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 46.87 2024-08-08 23:29:06,845 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 47.22 2024-08-08 23:29:16,977 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 50.21 2024-08-08 23:29:27,113 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: running, progress: 67.13 2024-08-08 23:29:42,296 | test | INFO | ThreadPool_0 | [task:check:912] Rebalance - status: none, progress: 100 2024-08-08 23:29:42,388 | test | INFO | ThreadPool_0 | [task:check:980] Rebalance completed with progress: 100% in 233.67549991607666 sec 2024-08-08 23:29:42,558 | test | CRITICAL | ThreadPool_0 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.74 2024-08-08 23:29:42,670 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] Cluster nodes..: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091', '172.23.108.70:18091', '172.23.108.72:18091', '172.23.108.74:18091'] 2024-08-08 23:29:42,670 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] KV............: ['172.23.108.67:18091', '172.23.108.68:18091'] 2024-08-08 23:29:42,671 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] Index.........: ['172.23.108.69:18091'] 2024-08-08 23:29:42,671 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] Query.........: ['172.23.108.69:18091'] 2024-08-08 23:29:42,671 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] CBAS..........: ['172.23.108.72:18091'] 2024-08-08 23:29:42,671 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] FTS...........: ['172.23.108.70:18091'] 2024-08-08 23:29:42,672 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] Eventing......: ['172.23.108.72:18091'] 2024-08-08 23:29:42,672 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] Backup........: ['172.23.108.70:18091'] 2024-08-08 23:29:42,672 | infra | CRITICAL | ThreadPool_0 | [task:print_nodes:642] Serviceless...: ['172.23.108.74:18091'] 2024-08-08 23:29:42,720 | test | INFO | MainThread | [travel_sample_app:trigger_remap_service_rebalance:259] {'index': ['ns_1@172.23.108.72'], 'n1ql': ['ns_1@172.23.108.72'], 'eventing': ['ns_1@172.23.108.69'], 'cbas': ['ns_1@172.23.108.69'], 'backup': ['ns_1@172.23.108.74'], 'fts': ['ns_1@172.23.108.74']} 2024-08-08 23:29:42,882 | test | INFO | ThreadPool_1 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Waiting before starting rebalance 2024-08-08 23:29:53,047 | test | CRITICAL | ThreadPool_1 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.74 2024-08-08 23:29:53,202 | test | INFO | ThreadPool_1 | [table_view:display:72] Rebalance Overview +---------------+---------+----------------+---------------------------------+-------------------+--------------+-----------------------+ | Nodes | Zone | Services | Version / Config | CPU | Status | Membership / Recovery | +---------------+---------+----------------+---------------------------------+-------------------+--------------+-----------------------+ | 172.23.108.67 | Group 1 | kv | 8.0.0-1775-enterprise / default | 17.34999999403954 | Cluster node | active / none | | 172.23.108.68 | Group 1 | kv | 8.0.0-1775-enterprise / default | 14.59999999962747 | Cluster node | active / none | | 172.23.108.69 | Group 1 | index, n1ql | 8.0.0-1775-enterprise / default | 14.14141414178671 | Cluster node | active / none | | 172.23.108.70 | Group 1 | backup, fts | 8.0.0-1775-enterprise / default | 12.1999999973923 | Cluster node | active / none | | 172.23.108.72 | Group 1 | eventing, cbas | 8.0.0-1775-enterprise / default | 64.70000000670552 | Cluster node | active / none | | 172.23.108.74 | Group 1 | | 8.0.0-1775-enterprise / default | 3.975000008940697 | Cluster node | active / none | +---------------+---------+----------------+---------------------------------+-------------------+--------------+-----------------------+ 2024-08-08 23:29:53,304 | test | INFO | ThreadPool_1 | [common_lib:sleep:21] Sleep 10 seconds. Reason: Wait before fetching rebalance stats 2024-08-08 23:30:03,370 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 41.5 2024-08-08 23:30:13,495 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 41.5 2024-08-08 23:30:23,615 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 41.5 2024-08-08 23:30:33,730 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 41.92 2024-08-08 23:30:43,845 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 42.31 2024-08-08 23:30:53,967 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 43.08 2024-08-08 23:31:04,097 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 43.46 2024-08-08 23:31:14,225 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 44.23 2024-08-08 23:31:24,340 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 45.0 2024-08-08 23:31:34,454 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 45.38 2024-08-08 23:31:44,579 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 46.15 2024-08-08 23:31:54,714 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 46.54 2024-08-08 23:32:04,829 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 47.31 2024-08-08 23:32:14,941 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 48.08 2024-08-08 23:32:25,057 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 48.46 2024-08-08 23:32:35,173 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 49.23 2024-08-08 23:32:45,287 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 49.62 2024-08-08 23:32:55,399 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 52.87 2024-08-08 23:33:05,513 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 54.61 2024-08-08 23:33:15,643 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 43.06 2024-08-08 23:33:25,755 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 52.68 2024-08-08 23:33:35,874 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: running, progress: 63.89 2024-08-08 23:33:51,016 | test | INFO | ThreadPool_1 | [task:check:912] Rebalance - status: none, progress: 100 2024-08-08 23:33:51,073 | test | INFO | ThreadPool_1 | [task:check:980] Rebalance completed with progress: 100% in 237.87093830108643 sec 2024-08-08 23:33:51,124 | test | INFO | ThreadPool_1 | [common_lib:sleep:21] Sleep 2 seconds. Reason: orchestrator='undefined' 2024-08-08 23:33:53,291 | test | CRITICAL | ThreadPool_1 | [cluster_ready_functions:validate_orchestrator_selection:516] Orchestrator: 172.23.108.70 2024-08-08 23:33:53,432 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] Cluster nodes..: ['172.23.108.67:18091', '172.23.108.68:18091', '172.23.108.69:18091', '172.23.108.70:18091', '172.23.108.72:18091', '172.23.108.74:18091'] 2024-08-08 23:33:53,432 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] KV............: ['172.23.108.67:18091', '172.23.108.68:18091'] 2024-08-08 23:33:53,432 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] Index.........: ['172.23.108.72:18091'] 2024-08-08 23:33:53,433 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] Query.........: ['172.23.108.72:18091'] 2024-08-08 23:33:53,433 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] CBAS..........: ['172.23.108.69:18091'] 2024-08-08 23:33:53,433 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] FTS...........: ['172.23.108.74:18091'] 2024-08-08 23:33:53,433 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] Eventing......: ['172.23.108.69:18091'] 2024-08-08 23:33:53,434 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] Backup........: ['172.23.108.74:18091'] 2024-08-08 23:33:53,434 | infra | CRITICAL | ThreadPool_1 | [task:print_nodes:642] Serviceless...: ['172.23.108.70:18091'] 2024-08-08 23:33:53,434 | test | INFO | MainThread | [app_basetest:__print_step:102] ############################################################ 5. Closing SDK client connections ############################################################ 2024-08-08 23:33:54,756 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:851] 172.23.108.67 - Parsing logs for error/critical string patterns 2024-08-08 23:33:55,527 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:33:56,200 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.67 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:33:56,945 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ['grep: /opt/couchbase/var/lib/couchbase/logs/memcached.log.000162.txt: No such file or directory'] ... 2024-08-08 23:33:57,005 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ['grep: /opt/couchbase/var/lib/couchbase/logs/memcached.log.000162.txt: No such file or directory'] ... 2024-08-08 23:33:57,061 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ['grep: /opt/couchbase/var/lib/couchbase/logs/memcached.log.000162.txt: No such file or directory'] ... 2024-08-08 23:34:12,359 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-08-08 23:34:13,068 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-08-08 23:34:24,263 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:851] 172.23.108.68 - Parsing logs for error/critical string patterns 2024-08-08 23:34:24,534 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:34:24,833 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.68 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:34:40,625 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-08-08 23:34:41,320 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-08-08 23:34:52,263 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:851] 172.23.108.69 - Parsing logs for error/critical string patterns 2024-08-08 23:34:52,538 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:34:52,861 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:34:53,126 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:34:53,441 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.69 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:34:59,418 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-08-08 23:35:24,044 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:851] 172.23.108.70 - Parsing logs for error/critical string patterns 2024-08-08 23:35:24,299 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:35:24,603 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:35:24,843 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:35:25,149 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.70 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:35:31,043 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-08-08 23:35:31,626 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... 2024-08-08 23:35:48,806 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:851] 172.23.108.72 - Parsing logs for error/critical string patterns 2024-08-08 23:35:49,072 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:35:49,383 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.72 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:35:49,643 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:35:49,944 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.72 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:35:55,787 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-08-08 23:36:15,862 | test | INFO | MainThread | [onPrem_basetestcase:check_coredump_exist:851] 172.23.108.74 - Parsing logs for error/critical string patterns 2024-08-08 23:36:16,058 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:36:16,324 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.74 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:36:16,526 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:363] os_distro: Ubuntu, os_version: debian 10, is_linux_distro: True 2024-08-08 23:36:16,786 | shell_util | INFO | MainThread | [shell_conn:extract_remote_info:519] 172.23.108.74 - distribution_type: Ubuntu, distribution_version: debian 10 2024-08-08 23:36:22,385 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/sanitizers.log.*': No such file or directory"] ... 2024-08-08 23:36:22,957 | shell_util | INFO | MainThread | [common_api:execute_command_raw:734] command executed with root but got an error ["ls: cannot access '/opt/couchbase/var/lib/couchbase/logs/indexer.log*': No such file or directory"] ... ok ---------------------------------------------------------------------- Ran 1 test in 1188.080s OK During the test, Remote Connections: 0, Disconnections: 0 SDK Connections: 15, Disconnections: 15 Summary:: suite bucket_collections.travel_sample_app.TravelSampleApp, pass 1, fail 0, scheduled 0 testrunner logs, diags and results are available under /data/workspace/temp_job_ashwin/logs/testrunner-24-Aug-08_23-16-44/test_1 + status=0 + exit 0 Archiving artifacts Recording test results [Checks API] No suitable checks publisher found. Finished: SUCCESS